mirror of
https://github.com/telemt/telemt.git
synced 2026-04-15 01:24:09 +03:00
Compare commits
95 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
12e68f805f | ||
|
|
62bf31fc73 | ||
|
|
29d4636249 | ||
|
|
9afaa28add | ||
|
|
6c12af2b94 | ||
|
|
8b39a4ef6d | ||
|
|
fa2423dadf | ||
|
|
449a87d2e3 | ||
|
|
a61882af6e | ||
|
|
bf11ebbaa3 | ||
|
|
e0d5561095 | ||
|
|
6b8aa7270e | ||
|
|
372f477927 | ||
|
|
05edbab06c | ||
|
|
3d9660f83e | ||
|
|
ac064fe773 | ||
|
|
eba158ff8b | ||
|
|
54ee6ff810 | ||
|
|
6d6cd30227 | ||
|
|
60231224ac | ||
|
|
144f81c473 | ||
|
|
04e6135935 | ||
|
|
4eebb4feb2 | ||
|
|
1f255d0aa4 | ||
|
|
9d2ff25bf5 | ||
|
|
7782336264 | ||
|
|
92a3529733 | ||
|
|
8ce8348cd5 | ||
|
|
e25b7f5ff8 | ||
|
|
d7182ae817 | ||
|
|
97f2dc8489 | ||
|
|
fb1f85559c | ||
|
|
da684b11fe | ||
|
|
896e129155 | ||
|
|
7ead0cd753 | ||
|
|
6cf9687dd6 | ||
|
|
4e30a4999c | ||
|
|
4af40f7121 | ||
|
|
1e4ba2eb56 | ||
|
|
eb921e2b17 | ||
|
|
76f1b51018 | ||
|
|
03ce267865 | ||
|
|
a6bfa3309e | ||
|
|
79a3720fd5 | ||
|
|
89543aed35 | ||
|
|
06292ff833 | ||
|
|
427294b103 | ||
|
|
fed9346444 | ||
|
|
f40b645c05 | ||
|
|
a66d5d56bb | ||
|
|
1b1bdfe99a | ||
|
|
49fc11ddfa | ||
|
|
5558900c44 | ||
|
|
5b1d976392 | ||
|
|
206f87fe64 | ||
|
|
5a09d30e1c | ||
|
|
f83e23c521 | ||
|
|
f9e9ddd0f7 | ||
|
|
6b8619d3c9 | ||
|
|
618b7a1837 | ||
|
|
16f166cec8 | ||
|
|
6efcbe9bbf | ||
|
|
e5ad27e26e | ||
|
|
53ec96b040 | ||
|
|
c6c3d71b08 | ||
|
|
e9a4281015 | ||
|
|
866c2fbd96 | ||
|
|
086c85d851 | ||
|
|
ce4e21c996 | ||
|
|
25ab79406f | ||
|
|
7538967d3c | ||
|
|
4a95f6d195 | ||
|
|
7d7ef84868 | ||
|
|
692d9476b9 | ||
|
|
b00b87032b | ||
|
|
ee07325eba | ||
|
|
1b3a17aedc | ||
|
|
6fdb568381 | ||
|
|
bb97ff0df9 | ||
|
|
b1cd7f9727 | ||
|
|
c13c1cf7e3 | ||
|
|
d2f08fb707 | ||
|
|
2356ae5584 | ||
|
|
429fa63c95 | ||
|
|
50e15896b3 | ||
|
|
09f56dede2 | ||
|
|
d9ae7bb044 | ||
|
|
d6214c6bbf | ||
|
|
3d3ddd37d7 | ||
|
|
1d71b7e90c | ||
|
|
8ba7bc9052 | ||
|
|
3397d82924 | ||
|
|
78c45626e1 | ||
|
|
68c3abee6c | ||
|
|
267c8bf2f1 |
2
.gitignore
vendored
2
.gitignore
vendored
@@ -19,3 +19,5 @@ target
|
||||
# and can be added to the global gitignore or merged into this file. For a more nuclear
|
||||
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
|
||||
#.idea/
|
||||
|
||||
proxy-secret
|
||||
|
||||
2
Cargo.lock
generated
2
Cargo.lock
generated
@@ -2087,7 +2087,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "telemt"
|
||||
version = "3.0.10"
|
||||
version = "3.0.13"
|
||||
dependencies = [
|
||||
"aes",
|
||||
"anyhow",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "telemt"
|
||||
version = "3.0.13"
|
||||
version = "3.1.3"
|
||||
edition = "2024"
|
||||
|
||||
[dependencies]
|
||||
|
||||
48
README.md
48
README.md
@@ -2,6 +2,8 @@
|
||||
|
||||
**Telemt** is a fast, secure, and feature-rich server written in Rust: it fully implements the official Telegram proxy algo and adds many production-ready improvements such as connection pooling, replay protection, detailed statistics, masking from "prying" eyes
|
||||
|
||||
[**Telemt Chat in Telegram**](https://t.me/telemtrs)
|
||||
|
||||
## NEWS and EMERGENCY
|
||||
### ✈️ Telemt 3 is released!
|
||||
<table>
|
||||
@@ -10,28 +12,18 @@
|
||||
|
||||
### 🇷🇺 RU
|
||||
|
||||
#### Драфтинг LTS и текущие улучшения
|
||||
#### Релиз 3.0.15 — 25 февраля
|
||||
|
||||
С 21 февраля мы начали подготовку LTS-версии.
|
||||
25 февраля мы выпустили версию **3.0.15**
|
||||
|
||||
Мы внимательно анализируем весь доступный фидбек.
|
||||
Наша цель — сделать LTS-кандидаты максимально стабильными, тщательно отлаженными и готовыми к long-run и highload production-сценариям.
|
||||
Мы предполагаем, что она станет завершающей версией поколения 3.0 и уже сейчас мы рассматриваем её как **LTS-кандидата** для версии **3.1.0**!
|
||||
|
||||
---
|
||||
После нескольких дней детального анализа особенностей работы Middle-End мы спроектировали и реализовали продуманный режим **ротации ME Writer**. Данный режим позволяет поддерживать стабильно высокую производительность в long-run сценариях без возникновения ошибок, связанных с некорректной конфигурацией прокси
|
||||
|
||||
#### Улучшения от 23 февраля
|
||||
|
||||
23 февраля были внесены улучшения производительности в режимах **DC** и **Middle-End (ME)**, с акцентом на обратный канал (путь клиент → DC / ME).
|
||||
|
||||
Дополнительно реализован ряд изменений, направленных на повышение устойчивости системы:
|
||||
|
||||
- Смягчение сетевой нестабильности
|
||||
- Повышение устойчивости к десинхронизации криптографии
|
||||
- Снижение дрейфа сессий при неблагоприятных условиях
|
||||
- Улучшение обработки ошибок в edge-case транспортных сценариях
|
||||
Будем рады вашему фидбеку и предложениям по улучшению — особенно в части **статистики** и **UX**
|
||||
|
||||
Релиз:
|
||||
[3.0.12](https://github.com/telemt/telemt/releases/tag/3.0.12)
|
||||
[3.0.15](https://github.com/telemt/telemt/releases/tag/3.0.15)
|
||||
|
||||
---
|
||||
|
||||
@@ -48,28 +40,18 @@
|
||||
|
||||
### 🇬🇧 EN
|
||||
|
||||
#### LTS Drafting and Ongoing Improvements
|
||||
#### Release 3.0.15 — February 25
|
||||
|
||||
Starting February 21, we began drafting the upcoming LTS version.
|
||||
On February 25, we released version **3.0.15**
|
||||
|
||||
We are carefully reviewing and analyzing all available feedback.
|
||||
The goal is to ensure that LTS candidates are максимально stable, thoroughly debugged, and ready for long-run and high-load production scenarios.
|
||||
We expect this to become the final release of the 3.0 generation and at this point, we already see it as a strong **LTS candidate** for the upcoming **3.1.0** release!
|
||||
|
||||
---
|
||||
After several days of deep analysis of Middle-End behavior, we designed and implemented a well-engineered **ME Writer rotation mode**. This mode enables sustained high throughput in long-run scenarios while preventing proxy misconfiguration errors
|
||||
|
||||
#### February 23 Improvements
|
||||
|
||||
On February 23, we introduced performance improvements for both **DC** and **Middle-End (ME)** modes, specifically optimizing the reverse channel (client → DC / ME data path).
|
||||
|
||||
Additionally, we implemented a set of robustness enhancements designed to:
|
||||
|
||||
- Mitigate network-related instability
|
||||
- Improve resilience against cryptographic desynchronization
|
||||
- Reduce session drift under adverse conditions
|
||||
- Improve error handling in edge-case transport scenarios
|
||||
We are looking forward to your feedback and improvement proposals — especially regarding **statistics** and **UX**
|
||||
|
||||
Release:
|
||||
[3.0.12](https://github.com/telemt/telemt/releases/tag/3.0.12)
|
||||
[3.0.15](https://github.com/telemt/telemt/releases/tag/3.0.15)
|
||||
|
||||
---
|
||||
|
||||
@@ -209,6 +191,8 @@ then Ctrl+X -> Y -> Enter to save
|
||||
|
||||
**5.** In Shell type `systemctl enable telemt` - then telemt will start with system startup, after the network is up
|
||||
|
||||
**6.** In Shell type `journalctl -u telemt -n -g "links" --no-pager -o cat | tac` - get the connection links
|
||||
|
||||
## Configuration
|
||||
### Minimal Configuration for First Start
|
||||
```toml
|
||||
|
||||
205
config.full.toml
Normal file
205
config.full.toml
Normal file
@@ -0,0 +1,205 @@
|
||||
# Telemt full config with default values.
|
||||
# Examples are kept in comments after '#'.
|
||||
|
||||
# Top-level legacy field.
|
||||
show_link = [] # example: "*" or ["alice", "bob"]
|
||||
# default_dc = 2 # example: default DC for unmapped non-standard DCs
|
||||
|
||||
[general]
|
||||
fast_mode = true
|
||||
use_middle_proxy = false
|
||||
# ad_tag = "00000000000000000000000000000000" # example
|
||||
# proxy_secret_path = "proxy-secret" # example custom path
|
||||
# middle_proxy_nat_ip = "203.0.113.10" # example public NAT IP override
|
||||
middle_proxy_nat_probe = true
|
||||
# middle_proxy_nat_stun = "stun.l.google.com:19302" # example
|
||||
# middle_proxy_nat_stun_servers = [] # example: ["stun1.l.google.com:19302", "stun2.l.google.com:19302"]
|
||||
middle_proxy_pool_size = 8
|
||||
middle_proxy_warm_standby = 16
|
||||
me_keepalive_enabled = true
|
||||
me_keepalive_interval_secs = 25
|
||||
me_keepalive_jitter_secs = 5
|
||||
me_keepalive_payload_random = true
|
||||
crypto_pending_buffer = 262144
|
||||
max_client_frame = 16777216
|
||||
desync_all_full = false
|
||||
beobachten = true
|
||||
beobachten_minutes = 10
|
||||
beobachten_flush_secs = 15
|
||||
beobachten_file = "cache/beobachten.txt"
|
||||
hardswap = true
|
||||
me_warmup_stagger_enabled = true
|
||||
me_warmup_step_delay_ms = 500
|
||||
me_warmup_step_jitter_ms = 300
|
||||
me_reconnect_max_concurrent_per_dc = 8
|
||||
me_reconnect_backoff_base_ms = 500
|
||||
me_reconnect_backoff_cap_ms = 30000
|
||||
me_reconnect_fast_retry_count = 12
|
||||
stun_iface_mismatch_ignore = false
|
||||
unknown_dc_log_path = "unknown-dc.txt" # to disable: set to null
|
||||
log_level = "normal" # debug | verbose | normal | silent
|
||||
disable_colors = false
|
||||
fast_mode_min_tls_record = 0
|
||||
update_every = 300
|
||||
me_reinit_every_secs = 900
|
||||
me_hardswap_warmup_delay_min_ms = 1000
|
||||
me_hardswap_warmup_delay_max_ms = 2000
|
||||
me_hardswap_warmup_extra_passes = 3
|
||||
me_hardswap_warmup_pass_backoff_base_ms = 500
|
||||
me_config_stable_snapshots = 2
|
||||
me_config_apply_cooldown_secs = 300
|
||||
proxy_secret_stable_snapshots = 2
|
||||
proxy_secret_rotate_runtime = true
|
||||
proxy_secret_len_max = 256
|
||||
me_pool_drain_ttl_secs = 90
|
||||
me_pool_min_fresh_ratio = 0.8
|
||||
me_reinit_drain_timeout_secs = 120
|
||||
# Legacy compatibility fields used when update_every is omitted.
|
||||
proxy_secret_auto_reload_secs = 3600
|
||||
proxy_config_auto_reload_secs = 3600
|
||||
ntp_check = true
|
||||
ntp_servers = ["pool.ntp.org"] # example: ["pool.ntp.org", "time.cloudflare.com"]
|
||||
auto_degradation_enabled = true
|
||||
degradation_min_unavailable_dc_groups = 2
|
||||
|
||||
[general.modes]
|
||||
classic = false
|
||||
secure = false
|
||||
tls = true
|
||||
|
||||
[general.links]
|
||||
show ="*" # example: "*" or ["alice", "bob"]
|
||||
# public_host = "proxy.example.com" # example explicit host/IP for tg:// links
|
||||
# public_port = 443 # example explicit port for tg:// links
|
||||
|
||||
[network]
|
||||
ipv4 = true
|
||||
ipv6 = false # set true to enable IPv6
|
||||
prefer = 4 # 4 or 6
|
||||
multipath = false
|
||||
stun_servers = [
|
||||
"stun.l.google.com:5349",
|
||||
"stun1.l.google.com:3478",
|
||||
"stun.gmx.net:3478",
|
||||
"stun.l.google.com:19302",
|
||||
"stun.1und1.de:3478",
|
||||
"stun1.l.google.com:19302",
|
||||
"stun2.l.google.com:19302",
|
||||
"stun3.l.google.com:19302",
|
||||
"stun4.l.google.com:19302",
|
||||
"stun.services.mozilla.com:3478",
|
||||
"stun.stunprotocol.org:3478",
|
||||
"stun.nextcloud.com:3478",
|
||||
"stun.voip.eutelia.it:3478",
|
||||
]
|
||||
stun_tcp_fallback = true
|
||||
http_ip_detect_urls = ["https://ifconfig.me/ip", "https://api.ipify.org"]
|
||||
cache_public_ip_path = "cache/public_ip.txt"
|
||||
|
||||
[server]
|
||||
port = 443
|
||||
listen_addr_ipv4 = "0.0.0.0"
|
||||
listen_addr_ipv6 = "::"
|
||||
# listen_unix_sock = "/var/run/telemt.sock" # example
|
||||
# listen_unix_sock_perm = "0660" # example unix socket mode
|
||||
# listen_tcp = true # example explicit override (auto-detected when omitted)
|
||||
proxy_protocol = false
|
||||
# metrics_port = 9090 # example
|
||||
metrics_whitelist = ["127.0.0.1/32", "::1/128"]
|
||||
# Example explicit listeners (default: omitted, auto-generated from listen_addr_*):
|
||||
# [[server.listeners]]
|
||||
# ip = "0.0.0.0"
|
||||
# announce = "proxy-v4.example.com"
|
||||
# # announce_ip = "203.0.113.10" # deprecated alias
|
||||
# proxy_protocol = false
|
||||
# reuse_allow = false
|
||||
#
|
||||
# [[server.listeners]]
|
||||
# ip = "::"
|
||||
# announce = "proxy-v6.example.com"
|
||||
# proxy_protocol = false
|
||||
# reuse_allow = false
|
||||
|
||||
[timeouts]
|
||||
client_handshake = 15
|
||||
tg_connect = 10
|
||||
client_keepalive = 60
|
||||
client_ack = 300
|
||||
me_one_retry = 3
|
||||
me_one_timeout_ms = 1500
|
||||
|
||||
[censorship]
|
||||
tls_domain = "petrovich.ru"
|
||||
# tls_domains = ["example.com", "cdn.example.net"] # Additional domains for EE links
|
||||
mask = true
|
||||
# mask_host = "www.google.com" # example, defaults to tls_domain when both mask_host/mask_unix_sock are unset
|
||||
# mask_unix_sock = "/var/run/nginx.sock" # example, mutually exclusive with mask_host
|
||||
mask_port = 443
|
||||
# mask_proxy_protocol = 0 # Send PROXY protocol header to mask_host: 0 = off, 1 = v1 (text), 2 = v2 (binary)
|
||||
fake_cert_len = 2048 # if tls_emulation=false and default value is used, loader may randomize this value at runtime
|
||||
tls_emulation = true
|
||||
tls_front_dir = "tlsfront"
|
||||
server_hello_delay_min_ms = 0
|
||||
server_hello_delay_max_ms = 0
|
||||
tls_new_session_tickets = 0
|
||||
tls_full_cert_ttl_secs = 90
|
||||
alpn_enforce = true
|
||||
|
||||
[access]
|
||||
replay_check_len = 65536
|
||||
replay_window_secs = 1800
|
||||
ignore_time_skew = false
|
||||
|
||||
[access.users]
|
||||
# format: "username" = "32_hex_chars_secret"
|
||||
hello = "00000000000000000000000000000000"
|
||||
# alice = "11111111111111111111111111111111" # example
|
||||
|
||||
[access.user_max_tcp_conns]
|
||||
# alice = 100 # example
|
||||
|
||||
[access.user_expirations]
|
||||
# alice = "2078-01-01T00:00:00Z" # example
|
||||
|
||||
[access.user_data_quota]
|
||||
# hello = 10737418240 # example bytes
|
||||
# alice = 10737418240 # example bytes
|
||||
|
||||
[access.user_max_unique_ips]
|
||||
# hello = 10 # example
|
||||
# alice = 100 # example
|
||||
|
||||
# Default behavior if [[upstreams]] is omitted: loader injects one direct upstream.
|
||||
# Example explicit upstreams:
|
||||
# [[upstreams]]
|
||||
# type = "direct"
|
||||
# interface = "eth0"
|
||||
# bind_addresses = ["192.0.2.10"]
|
||||
# weight = 1
|
||||
# enabled = true
|
||||
# scopes = "*"
|
||||
#
|
||||
# [[upstreams]]
|
||||
# type = "socks4"
|
||||
# address = "198.51.100.20:1080"
|
||||
# interface = "eth0"
|
||||
# user_id = "telemt"
|
||||
# weight = 1
|
||||
# enabled = true
|
||||
# scopes = "*"
|
||||
#
|
||||
# [[upstreams]]
|
||||
# type = "socks5"
|
||||
# address = "198.51.100.30:1080"
|
||||
# interface = "eth0"
|
||||
# username = "proxy-user"
|
||||
# password = "proxy-pass"
|
||||
# weight = 1
|
||||
# enabled = true
|
||||
# scopes = "*"
|
||||
|
||||
# === DC Address Overrides ===
|
||||
# [dc_overrides]
|
||||
# "201" = "149.154.175.50:443" # example
|
||||
# "202" = ["149.154.167.51:443", "149.154.175.100:443"] # example
|
||||
# "203" = "91.105.192.100:443" # loader auto-adds this one when omitted
|
||||
122
config.toml
122
config.toml
@@ -1,11 +1,11 @@
|
||||
### Telemt Based Config.toml
|
||||
# We believe that these settings are sufficient for most scenarios
|
||||
# where cutting-egde methods and parameters or special solutions are not needed
|
||||
|
||||
# === General Settings ===
|
||||
[general]
|
||||
fast_mode = true
|
||||
use_middle_proxy = true
|
||||
use_middle_proxy = false
|
||||
# ad_tag = "00000000000000000000000000000000"
|
||||
# Path to proxy-secret binary (auto-downloaded if missing).
|
||||
proxy_secret_path = "proxy-secret"
|
||||
# disable_colors = false # Disable colored output in logs (useful for files/systemd)
|
||||
|
||||
# === Log Level ===
|
||||
# Log level: debug | verbose | normal | silent
|
||||
@@ -13,43 +13,6 @@ proxy_secret_path = "proxy-secret"
|
||||
# RUST_LOG env var takes absolute priority over all of these
|
||||
log_level = "normal"
|
||||
|
||||
# === Middle Proxy - ME ===
|
||||
# Public IP override for ME KDF when behind NAT; leave unset to auto-detect.
|
||||
# middle_proxy_nat_ip = "203.0.113.10"
|
||||
# Enable STUN probing to discover public IP:port for ME.
|
||||
middle_proxy_nat_probe = true
|
||||
# Primary STUN server (host:port); defaults to Telegram STUN when empty.
|
||||
middle_proxy_nat_stun = "stun.l.google.com:19302"
|
||||
# Optional fallback STUN servers list.
|
||||
middle_proxy_nat_stun_servers = ["stun1.l.google.com:19302", "stun2.l.google.com:19302"]
|
||||
# Desired number of concurrent ME writers in pool.
|
||||
middle_proxy_pool_size = 16
|
||||
# Pre-initialized warm-standby ME connections kept idle.
|
||||
middle_proxy_warm_standby = 8
|
||||
# Ignore STUN/interface mismatch and keep ME enabled even if IP differs.
|
||||
stun_iface_mismatch_ignore = false
|
||||
# Keepalive padding frames - fl==4
|
||||
me_keepalive_enabled = true
|
||||
me_keepalive_interval_secs = 25 # Period between keepalives
|
||||
me_keepalive_jitter_secs = 5 # Jitter added to interval
|
||||
me_keepalive_payload_random = true # Randomize 4-byte payload (vs zeros)
|
||||
# Stagger extra ME connections on warmup to de-phase lifecycles.
|
||||
me_warmup_stagger_enabled = true
|
||||
me_warmup_step_delay_ms = 500 # Base delay between extra connects
|
||||
me_warmup_step_jitter_ms = 300 # Jitter for warmup delay
|
||||
# Reconnect policy knobs.
|
||||
me_reconnect_max_concurrent_per_dc = 4 # Parallel reconnects per DC - EXPERIMENTAL! UNSTABLE!
|
||||
me_reconnect_backoff_base_ms = 500 # Backoff start
|
||||
me_reconnect_backoff_cap_ms = 30000 # Backoff cap
|
||||
me_reconnect_fast_retry_count = 11 # Quick retries before backoff
|
||||
update_every = 7200 # Resolve the active updater interval for ME infrastructure refresh tasks.
|
||||
crypto_pending_buffer = 262144 # Max pending ciphertext buffer per client writer (bytes). Controls FakeTLS backpressure vs throughput.
|
||||
max_client_frame = 16777216 # Maximum allowed client MTProto frame size (bytes).
|
||||
desync_all_full = false # Emit full crypto-desync forensic logs for every event. When false, full forensic details are emitted once per key window.
|
||||
me_reinit_drain_timeout_secs = 300 # Drain timeout in seconds for stale ME writers after endpoint map changes. Set to 0 to keep stale writers draining indefinitely (no force-close).
|
||||
auto_degradation_enabled = true # Enable auto-degradation from ME to Direct-DC.
|
||||
degradation_min_unavailable_dc_groups = 2 # Minimum unavailable ME DC groups before degrading.
|
||||
|
||||
[general.modes]
|
||||
classic = false
|
||||
secure = false
|
||||
@@ -62,93 +25,24 @@ show = "*"
|
||||
# public_host = "proxy.example.com" # Host (IP or domain) for tg:// links
|
||||
# public_port = 443 # Port for tg:// links (default: server.port)
|
||||
|
||||
# === Network Parameters ===
|
||||
[network]
|
||||
# Enable/disable families: true/false/auto(None)
|
||||
ipv4 = true
|
||||
ipv6 = false # UNSTABLE WITH ME
|
||||
# prefer = 4 or 6
|
||||
prefer = 4
|
||||
multipath = false # EXPERIMENTAL!
|
||||
|
||||
# === Server Binding ===
|
||||
[server]
|
||||
port = 443
|
||||
listen_addr_ipv4 = "0.0.0.0"
|
||||
listen_addr_ipv6 = "::"
|
||||
# listen_unix_sock = "/var/run/telemt.sock" # Unix socket
|
||||
# listen_unix_sock_perm = "0666" # Socket file permissions
|
||||
# proxy_protocol = false # Enable if behind HAProxy/nginx with PROXY protocol
|
||||
# metrics_port = 9090
|
||||
# metrics_whitelist = ["127.0.0.1", "::1"]
|
||||
# metrics_whitelist = ["127.0.0.1", "::1", "0.0.0.0/0"]
|
||||
|
||||
# Listen on multiple interfaces/IPs - IPv4
|
||||
[[server.listeners]]
|
||||
ip = "0.0.0.0"
|
||||
|
||||
# Listen on multiple interfaces/IPs - IPv6
|
||||
[[server.listeners]]
|
||||
ip = "::"
|
||||
|
||||
# === Timeouts (in seconds) ===
|
||||
[timeouts]
|
||||
client_handshake = 30
|
||||
tg_connect = 10
|
||||
client_keepalive = 60
|
||||
client_ack = 300
|
||||
# Quick ME reconnects for single-address DCs (count and per-attempt timeout, ms).
|
||||
me_one_retry = 12
|
||||
me_one_timeout_ms = 1200
|
||||
|
||||
# === Anti-Censorship & Masking ===
|
||||
[censorship]
|
||||
tls_domain = "petrovich.ru"
|
||||
# tls_domains = ["example.com", "cdn.example.net"] # Additional domains for EE links
|
||||
mask = true
|
||||
mask_port = 443
|
||||
# mask_host = "petrovich.ru" # Defaults to tls_domain if not set
|
||||
# mask_unix_sock = "/var/run/nginx.sock" # Unix socket (mutually exclusive with mask_host)
|
||||
fake_cert_len = 2048
|
||||
# tls_emulation = false # Fetch real cert lengths and emulate TLS records
|
||||
# tls_front_dir = "tlsfront" # Cache directory for TLS emulation
|
||||
|
||||
# === Access Control & Users ===
|
||||
[access]
|
||||
replay_check_len = 65536
|
||||
replay_window_secs = 1800
|
||||
ignore_time_skew = false
|
||||
tls_emulation = true # Fetch real cert lengths and emulate TLS records
|
||||
tls_front_dir = "tlsfront" # Cache directory for TLS emulation
|
||||
|
||||
[access.users]
|
||||
# format: "username" = "32_hex_chars_secret"
|
||||
hello = "00000000000000000000000000000000"
|
||||
|
||||
# [access.user_max_tcp_conns]
|
||||
# hello = 50
|
||||
|
||||
# [access.user_max_unique_ips]
|
||||
# hello = 5
|
||||
|
||||
# [access.user_data_quota]
|
||||
# hello = 1073741824 # 1 GB
|
||||
|
||||
# [access.user_expirations]
|
||||
# format: username = "[year]-[month]-[day]T[hour]:[minute]:[second]Z" UTC
|
||||
# hello = "2027-01-01T00:00:00Z"
|
||||
|
||||
# === Upstreams & Routing ===
|
||||
[[upstreams]]
|
||||
type = "direct"
|
||||
enabled = true
|
||||
weight = 10
|
||||
# interface = "192.168.1.100" # Bind outgoing to specific IP or iface name
|
||||
# bind_addresses = ["192.168.1.100"] # List for round-robin binding (family must match target)
|
||||
|
||||
# [[upstreams]]
|
||||
# type = "socks5"
|
||||
# address = "127.0.0.1:1080"
|
||||
# enabled = false
|
||||
# weight = 1
|
||||
|
||||
# === DC Address Overrides ===
|
||||
# [dc_overrides]
|
||||
# "203" = "91.105.192.100:443"
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
services:
|
||||
telemt:
|
||||
image: ghcr.io/telemt/telemt:latest
|
||||
build: .
|
||||
container_name: telemt
|
||||
restart: unless-stopped
|
||||
|
||||
219
docs/TUNING.de.md
Normal file
219
docs/TUNING.de.md
Normal file
@@ -0,0 +1,219 @@
|
||||
# Telemt Tuning-Leitfaden: Middle-End und Upstreams
|
||||
|
||||
Dieses Dokument beschreibt das aktuelle Laufzeitverhalten für Middle-End (ME) und Upstream-Routing basierend auf:
|
||||
- `src/config/types.rs`
|
||||
- `src/config/defaults.rs`
|
||||
- `src/config/load.rs`
|
||||
- `src/transport/upstream.rs`
|
||||
|
||||
Die unten angegebenen `Default`-Werte sind Code-Defaults (bei fehlendem Schlüssel), nicht zwingend die Werte aus `config.full.toml`.
|
||||
|
||||
## Middle-End-Parameter
|
||||
|
||||
### 1) ME-Grundmodus, NAT und STUN
|
||||
|
||||
| Parameter | Typ | Default | Einschränkungen / Validierung | Laufzeiteffekt | Beispiel |
|
||||
|---|---|---:|---|---|---|
|
||||
| `general.use_middle_proxy` | `bool` | `true` | keine | Aktiviert den ME-Transportmodus. Bei `false` wird Direct-Modus verwendet. | `use_middle_proxy = true` |
|
||||
| `general.proxy_secret_path` | `Option<String>` | `"proxy-secret"` | Pfad kann `null` sein | Pfad zur Telegram-Infrastrukturdatei `proxy-secret`. | `proxy_secret_path = "proxy-secret"` |
|
||||
| `general.middle_proxy_nat_ip` | `Option<IpAddr>` | `null` | gültige IP bei gesetztem Wert | Manueller Override der öffentlichen NAT-IP für ME-Adressmaterial. | `middle_proxy_nat_ip = "203.0.113.10"` |
|
||||
| `general.middle_proxy_nat_probe` | `bool` | `true` | wird auf `true` erzwungen, wenn `use_middle_proxy=true` | Aktiviert NAT-Probing für ME. | `middle_proxy_nat_probe = true` |
|
||||
| `general.stun_nat_probe_concurrency` | `usize` | `8` | muss `> 0` sein | Maximale parallele STUN-Probes während NAT-Erkennung. | `stun_nat_probe_concurrency = 16` |
|
||||
| `network.stun_use` | `bool` | `true` | keine | Globaler STUN-Schalter. Bei `false` wird STUN deaktiviert. | `stun_use = true` |
|
||||
| `network.stun_servers` | `Vec<String>` | integrierter öffentlicher Pool | Duplikate/leer werden entfernt | Primäre STUN-Serverliste für NAT/Public-Endpoint-Erkennung. | `stun_servers = ["stun1.l.google.com:19302"]` |
|
||||
| `network.stun_tcp_fallback` | `bool` | `true` | keine | Aktiviert TCP-Fallback, wenn UDP-STUN blockiert ist. | `stun_tcp_fallback = true` |
|
||||
| `network.http_ip_detect_urls` | `Vec<String>` | `ifconfig.me` + `api.ipify.org` | keine | HTTP-Fallback zur öffentlichen IPv4-Erkennung, falls STUN ausfällt. | `http_ip_detect_urls = ["https://api.ipify.org"]` |
|
||||
| `general.stun_iface_mismatch_ignore` | `bool` | `false` | keine | Reserviertes Feld in der aktuellen Revision (derzeit kein aktiver Runtime-Verbrauch). | `stun_iface_mismatch_ignore = false` |
|
||||
| `timeouts.me_one_retry` | `u8` | `12` | keine | Anzahl schneller Reconnect-Versuche bei Single-Endpoint-DC-Fällen. | `me_one_retry = 6` |
|
||||
| `timeouts.me_one_timeout_ms` | `u64` | `1200` | keine | Timeout pro schnellem Einzelversuch (ms). | `me_one_timeout_ms = 1500` |
|
||||
|
||||
### 2) Poolgröße, Keepalive und Reconnect-Policy
|
||||
|
||||
| Parameter | Typ | Default | Einschränkungen / Validierung | Laufzeiteffekt | Beispiel |
|
||||
|---|---|---:|---|---|---|
|
||||
| `general.middle_proxy_pool_size` | `usize` | `8` | keine | Zielgröße des aktiven ME-Writer-Pools. | `middle_proxy_pool_size = 12` |
|
||||
| `general.middle_proxy_warm_standby` | `usize` | `16` | keine | Reserviertes Kompatibilitätsfeld in der aktuellen Revision (kein aktiver Runtime-Consumer). | `middle_proxy_warm_standby = 16` |
|
||||
| `general.me_keepalive_enabled` | `bool` | `true` | keine | Aktiviert periodischen ME-Keepalive/Ping-Traffic. | `me_keepalive_enabled = true` |
|
||||
| `general.me_keepalive_interval_secs` | `u64` | `25` | keine | Basisintervall für Keepalive (Sekunden). | `me_keepalive_interval_secs = 20` |
|
||||
| `general.me_keepalive_jitter_secs` | `u64` | `5` | keine | Keepalive-Jitter zur Vermeidung synchroner Peaks. | `me_keepalive_jitter_secs = 3` |
|
||||
| `general.me_keepalive_payload_random` | `bool` | `true` | keine | Randomisiert Keepalive-Payload-Bytes. | `me_keepalive_payload_random = true` |
|
||||
| `general.me_warmup_stagger_enabled` | `bool` | `true` | keine | Aktiviert gestaffeltes Warmup zusätzlicher ME-Verbindungen. | `me_warmup_stagger_enabled = true` |
|
||||
| `general.me_warmup_step_delay_ms` | `u64` | `500` | keine | Basisverzögerung zwischen Warmup-Schritten (ms). | `me_warmup_step_delay_ms = 300` |
|
||||
| `general.me_warmup_step_jitter_ms` | `u64` | `300` | keine | Zusätzlicher zufälliger Warmup-Jitter (ms). | `me_warmup_step_jitter_ms = 200` |
|
||||
| `general.me_reconnect_max_concurrent_per_dc` | `u32` | `8` | keine | Begrenzung paralleler Reconnect-Worker pro DC. | `me_reconnect_max_concurrent_per_dc = 12` |
|
||||
| `general.me_reconnect_backoff_base_ms` | `u64` | `500` | keine | Initiales Reconnect-Backoff (ms). | `me_reconnect_backoff_base_ms = 250` |
|
||||
| `general.me_reconnect_backoff_cap_ms` | `u64` | `30000` | keine | Maximales Reconnect-Backoff (ms). | `me_reconnect_backoff_cap_ms = 10000` |
|
||||
| `general.me_reconnect_fast_retry_count` | `u32` | `16` | keine | Budget für Sofort-Retries vor längerem Backoff. | `me_reconnect_fast_retry_count = 8` |
|
||||
|
||||
### 3) Reinit/Hardswap, Secret-Rotation und Degradation
|
||||
|
||||
| Parameter | Typ | Default | Einschränkungen / Validierung | Laufzeiteffekt | Beispiel |
|
||||
|---|---|---:|---|---|---|
|
||||
| `general.hardswap` | `bool` | `true` | keine | Aktiviert generation-basierte Hardswap-Strategie für den ME-Pool. | `hardswap = true` |
|
||||
| `general.me_reinit_every_secs` | `u64` | `900` | muss `> 0` sein | Intervall für periodische ME-Reinitialisierung. | `me_reinit_every_secs = 600` |
|
||||
| `general.me_hardswap_warmup_delay_min_ms` | `u64` | `1000` | muss `<= me_hardswap_warmup_delay_max_ms` sein | Untere Grenze für Warmup-Dial-Abstände. | `me_hardswap_warmup_delay_min_ms = 500` |
|
||||
| `general.me_hardswap_warmup_delay_max_ms` | `u64` | `2000` | muss `> 0` sein | Obere Grenze für Warmup-Dial-Abstände. | `me_hardswap_warmup_delay_max_ms = 1200` |
|
||||
| `general.me_hardswap_warmup_extra_passes` | `u8` | `3` | Bereich `[0,10]` | Zusätzliche Warmup-Pässe nach dem Basispass. | `me_hardswap_warmup_extra_passes = 2` |
|
||||
| `general.me_hardswap_warmup_pass_backoff_base_ms` | `u64` | `500` | muss `> 0` sein | Basis-Backoff zwischen zusätzlichen Warmup-Pässen. | `me_hardswap_warmup_pass_backoff_base_ms = 400` |
|
||||
| `general.me_config_stable_snapshots` | `u8` | `2` | muss `> 0` sein | Anzahl identischer ME-Config-Snapshots vor Apply. | `me_config_stable_snapshots = 3` |
|
||||
| `general.me_config_apply_cooldown_secs` | `u64` | `300` | keine | Cooldown zwischen angewendeten ME-Map-Updates. | `me_config_apply_cooldown_secs = 120` |
|
||||
| `general.proxy_secret_stable_snapshots` | `u8` | `2` | muss `> 0` sein | Anzahl identischer Secret-Snapshots vor Rotation. | `proxy_secret_stable_snapshots = 3` |
|
||||
| `general.proxy_secret_rotate_runtime` | `bool` | `true` | keine | Aktiviert Runtime-Rotation des Proxy-Secrets. | `proxy_secret_rotate_runtime = true` |
|
||||
| `general.proxy_secret_len_max` | `usize` | `256` | Bereich `[32,4096]` | Obergrenze für akzeptierte Secret-Länge. | `proxy_secret_len_max = 512` |
|
||||
| `general.update_every` | `Option<u64>` | `300` | wenn gesetzt: `> 0`; bei `null`: Legacy-Min-Fallback | Einheitliches Refresh-Intervall für ME-Config + Secret-Updater. | `update_every = 300` |
|
||||
| `general.me_pool_drain_ttl_secs` | `u64` | `90` | keine | Zeitraum, in dem stale Writer noch als Fallback zulässig sind. | `me_pool_drain_ttl_secs = 120` |
|
||||
| `general.me_pool_min_fresh_ratio` | `f32` | `0.8` | Bereich `[0.0,1.0]` | Coverage-Schwelle vor Drain der alten Generation. | `me_pool_min_fresh_ratio = 0.9` |
|
||||
| `general.me_reinit_drain_timeout_secs` | `u64` | `120` | `0` = kein Force-Close; wenn `>0 && < TTL`, dann auf TTL angehoben | Force-Close-Timeout für draining stale Writer. | `me_reinit_drain_timeout_secs = 0` |
|
||||
| `general.auto_degradation_enabled` | `bool` | `true` | keine | Reserviertes Kompatibilitätsfeld in aktueller Revision (kein aktiver Runtime-Consumer). | `auto_degradation_enabled = true` |
|
||||
| `general.degradation_min_unavailable_dc_groups` | `u8` | `2` | keine | Reservierter Kompatibilitäts-Schwellenwert in aktueller Revision (kein aktiver Runtime-Consumer). | `degradation_min_unavailable_dc_groups = 2` |
|
||||
|
||||
## Deprecated / Legacy Parameter
|
||||
|
||||
| Parameter | Status | Ersatz | Aktuelles Verhalten | Migrationshinweis |
|
||||
|---|---|---|---|---|
|
||||
| `general.middle_proxy_nat_stun` | Deprecated | `network.stun_servers` | Wird nur dann in `network.stun_servers` gemerged, wenn `network.stun_servers` nicht explizit gesetzt ist. | Wert nach `network.stun_servers` verschieben, Legacy-Key entfernen. |
|
||||
| `general.middle_proxy_nat_stun_servers` | Deprecated | `network.stun_servers` | Wird nur dann in `network.stun_servers` gemerged, wenn `network.stun_servers` nicht explizit gesetzt ist. | Werte nach `network.stun_servers` verschieben, Legacy-Key entfernen. |
|
||||
| `general.proxy_secret_auto_reload_secs` | Deprecated | `general.update_every` | Nur aktiv, wenn `update_every = null` (Legacy-Fallback). | `general.update_every` explizit setzen, Legacy-Key entfernen. |
|
||||
| `general.proxy_config_auto_reload_secs` | Deprecated | `general.update_every` | Nur aktiv, wenn `update_every = null` (Legacy-Fallback). | `general.update_every` explizit setzen, Legacy-Key entfernen. |
|
||||
|
||||
## Wie Upstreams konfiguriert werden
|
||||
|
||||
### Upstream-Schema
|
||||
|
||||
| Feld | Gilt für | Typ | Pflicht | Default | Bedeutung |
|
||||
|---|---|---|---|---|---|
|
||||
| `[[upstreams]].type` | alle Upstreams | `"direct" \| "socks4" \| "socks5"` | ja | n/a | Upstream-Transporttyp. |
|
||||
| `[[upstreams]].weight` | alle Upstreams | `u16` | nein | `1` | Basisgewicht für weighted-random Auswahl. |
|
||||
| `[[upstreams]].enabled` | alle Upstreams | `bool` | nein | `true` | Deaktivierte Einträge werden beim Start ignoriert. |
|
||||
| `[[upstreams]].scopes` | alle Upstreams | `String` | nein | `""` | Komma-separierte Scope-Tags für Request-Routing. |
|
||||
| `interface` | `direct` | `Option<String>` | nein | `null` | Interface-Name (z. B. `eth0`) oder lokale Literal-IP. |
|
||||
| `bind_addresses` | `direct` | `Option<Vec<IpAddr>>` | nein | `null` | Explizite Source-IP-Kandidaten (strikter Vorrang vor `interface`). |
|
||||
| `address` | `socks4` | `String` | ja | n/a | SOCKS4-Server (`ip:port` oder `host:port`). |
|
||||
| `interface` | `socks4` | `Option<String>` | nein | `null` | Wird nur genutzt, wenn `address` als `ip:port` angegeben ist. |
|
||||
| `user_id` | `socks4` | `Option<String>` | nein | `null` | SOCKS4 User-ID für CONNECT. |
|
||||
| `address` | `socks5` | `String` | ja | n/a | SOCKS5-Server (`ip:port` oder `host:port`). |
|
||||
| `interface` | `socks5` | `Option<String>` | nein | `null` | Wird nur genutzt, wenn `address` als `ip:port` angegeben ist. |
|
||||
| `username` | `socks5` | `Option<String>` | nein | `null` | SOCKS5 Benutzername. |
|
||||
| `password` | `socks5` | `Option<String>` | nein | `null` | SOCKS5 Passwort. |
|
||||
|
||||
### Runtime-Regeln (wichtig)
|
||||
|
||||
1. Wenn `[[upstreams]]` fehlt, injiziert der Loader einen Default-`direct`-Upstream.
|
||||
2. Scope-Filterung basiert auf exaktem Token-Match:
|
||||
- mit Request-Scope -> nur Einträge, deren `scopes` genau dieses Token enthält;
|
||||
- ohne Request-Scope -> nur Einträge mit leerem `scopes`.
|
||||
3. Unter healthy Upstreams erfolgt die Auswahl per weighted random: `weight * latency_factor`.
|
||||
4. Gibt es im gefilterten Set keinen healthy Upstream, wird zufällig aus dem gefilterten Set gewählt.
|
||||
5. `direct`-Bind-Auflösung:
|
||||
- zuerst `bind_addresses` (nur gleiche IP-Familie wie Target);
|
||||
- bei `interface` (Name) + `bind_addresses` wird jede Candidate-IP gegen Interface-Adressen validiert;
|
||||
- ungültige Kandidaten werden mit `WARN` verworfen;
|
||||
- bleiben keine gültigen Kandidaten übrig, erfolgt unbound direct connect (`bind_ip=None`);
|
||||
- wenn `bind_addresses` nicht passt, wird `interface` verwendet (Literal-IP oder Interface-Primäradresse).
|
||||
6. Für `socks4/socks5` mit Hostname-`address` ist Interface-Binding nicht unterstützt und wird mit Warnung ignoriert.
|
||||
7. Runtime DNS Overrides werden für Hostname-Auflösung bei Upstream-Verbindungen genutzt.
|
||||
8. Im ME-Modus wird der gewählte Upstream auch für den ME-TCP-Dial-Pfad verwendet.
|
||||
9. Im ME-Modus ist bei `direct` mit bind/interface die STUN-Reflection bind-aware für KDF-Adressmaterial.
|
||||
10. Im ME-Modus werden bei SOCKS-Upstream `BND.ADDR/BND.PORT` für KDF verwendet, wenn gültig/öffentlich und gleiche IP-Familie.
|
||||
|
||||
## Upstream-Konfigurationsbeispiele
|
||||
|
||||
### Beispiel 1: Minimaler direct Upstream
|
||||
|
||||
```toml
|
||||
[[upstreams]]
|
||||
type = "direct"
|
||||
weight = 1
|
||||
enabled = true
|
||||
```
|
||||
|
||||
### Beispiel 2: direct mit Interface + expliziten bind IPs
|
||||
|
||||
```toml
|
||||
[[upstreams]]
|
||||
type = "direct"
|
||||
interface = "eth0"
|
||||
bind_addresses = ["192.168.1.100", "192.168.1.101"]
|
||||
weight = 3
|
||||
enabled = true
|
||||
```
|
||||
|
||||
### Beispiel 3: SOCKS5 Upstream mit Authentifizierung
|
||||
|
||||
```toml
|
||||
[[upstreams]]
|
||||
type = "socks5"
|
||||
address = "198.51.100.30:1080"
|
||||
username = "proxy-user"
|
||||
password = "proxy-pass"
|
||||
weight = 2
|
||||
enabled = true
|
||||
```
|
||||
|
||||
### Beispiel 4: Gemischte Upstreams mit Scopes
|
||||
|
||||
```toml
|
||||
[[upstreams]]
|
||||
type = "direct"
|
||||
weight = 5
|
||||
enabled = true
|
||||
scopes = ""
|
||||
|
||||
[[upstreams]]
|
||||
type = "socks5"
|
||||
address = "203.0.113.40:1080"
|
||||
username = "edge"
|
||||
password = "edgepass"
|
||||
weight = 3
|
||||
enabled = true
|
||||
scopes = "premium,me"
|
||||
```
|
||||
|
||||
### Beispiel 5: ME-orientiertes Tuning-Profil
|
||||
|
||||
```toml
|
||||
[general]
|
||||
use_middle_proxy = true
|
||||
proxy_secret_path = "proxy-secret"
|
||||
middle_proxy_nat_probe = true
|
||||
stun_nat_probe_concurrency = 16
|
||||
middle_proxy_pool_size = 12
|
||||
me_keepalive_enabled = true
|
||||
me_keepalive_interval_secs = 20
|
||||
me_keepalive_jitter_secs = 4
|
||||
me_reconnect_max_concurrent_per_dc = 12
|
||||
me_reconnect_backoff_base_ms = 300
|
||||
me_reconnect_backoff_cap_ms = 10000
|
||||
me_reconnect_fast_retry_count = 10
|
||||
hardswap = true
|
||||
me_reinit_every_secs = 600
|
||||
me_hardswap_warmup_delay_min_ms = 500
|
||||
me_hardswap_warmup_delay_max_ms = 1200
|
||||
me_hardswap_warmup_extra_passes = 2
|
||||
me_hardswap_warmup_pass_backoff_base_ms = 400
|
||||
me_config_stable_snapshots = 3
|
||||
me_config_apply_cooldown_secs = 120
|
||||
proxy_secret_stable_snapshots = 3
|
||||
proxy_secret_rotate_runtime = true
|
||||
proxy_secret_len_max = 512
|
||||
update_every = 300
|
||||
me_pool_drain_ttl_secs = 120
|
||||
me_pool_min_fresh_ratio = 0.9
|
||||
me_reinit_drain_timeout_secs = 180
|
||||
|
||||
[timeouts]
|
||||
me_one_retry = 8
|
||||
me_one_timeout_ms = 1200
|
||||
|
||||
[network]
|
||||
stun_use = true
|
||||
stun_tcp_fallback = true
|
||||
stun_servers = [
|
||||
"stun1.l.google.com:19302",
|
||||
"stun2.l.google.com:19302"
|
||||
]
|
||||
http_ip_detect_urls = [
|
||||
"https://api.ipify.org",
|
||||
"https://ifconfig.me/ip"
|
||||
]
|
||||
```
|
||||
219
docs/TUNING.en.md
Normal file
219
docs/TUNING.en.md
Normal file
@@ -0,0 +1,219 @@
|
||||
# Telemt Tuning Guide: Middle-End and Upstreams
|
||||
|
||||
This document describes the current runtime behavior for Middle-End (ME) and upstream routing based on:
|
||||
- `src/config/types.rs`
|
||||
- `src/config/defaults.rs`
|
||||
- `src/config/load.rs`
|
||||
- `src/transport/upstream.rs`
|
||||
|
||||
Defaults below are code defaults (used when a key is omitted), not necessarily values from `config.full.toml` examples.
|
||||
|
||||
## Middle-End Parameters
|
||||
|
||||
### 1) Core ME mode, NAT, and STUN
|
||||
|
||||
| Parameter | Type | Default | Constraints / validation | Runtime effect | Example |
|
||||
|---|---|---:|---|---|---|
|
||||
| `general.use_middle_proxy` | `bool` | `true` | none | Enables ME transport mode. If `false`, Direct mode is used. | `use_middle_proxy = true` |
|
||||
| `general.proxy_secret_path` | `Option<String>` | `"proxy-secret"` | path may be `null` | Path to Telegram infrastructure proxy-secret file. | `proxy_secret_path = "proxy-secret"` |
|
||||
| `general.middle_proxy_nat_ip` | `Option<IpAddr>` | `null` | valid IP when set | Manual public NAT IP override for ME address material. | `middle_proxy_nat_ip = "203.0.113.10"` |
|
||||
| `general.middle_proxy_nat_probe` | `bool` | `true` | auto-forced to `true` when `use_middle_proxy=true` | Enables ME NAT probing. | `middle_proxy_nat_probe = true` |
|
||||
| `general.stun_nat_probe_concurrency` | `usize` | `8` | must be `> 0` | Max parallel STUN probes during NAT discovery. | `stun_nat_probe_concurrency = 16` |
|
||||
| `network.stun_use` | `bool` | `true` | none | Global STUN switch. If `false`, STUN probing is disabled. | `stun_use = true` |
|
||||
| `network.stun_servers` | `Vec<String>` | built-in public pool | deduplicated + empty values removed | Primary STUN server list for NAT/public endpoint discovery. | `stun_servers = ["stun1.l.google.com:19302"]` |
|
||||
| `network.stun_tcp_fallback` | `bool` | `true` | none | Enables TCP fallback path when UDP STUN is blocked. | `stun_tcp_fallback = true` |
|
||||
| `network.http_ip_detect_urls` | `Vec<String>` | `ifconfig.me` + `api.ipify.org` | none | HTTP fallback for public IPv4 detection if STUN is unavailable. | `http_ip_detect_urls = ["https://api.ipify.org"]` |
|
||||
| `general.stun_iface_mismatch_ignore` | `bool` | `false` | none | Reserved flag in current revision (not consumed by runtime path). | `stun_iface_mismatch_ignore = false` |
|
||||
| `timeouts.me_one_retry` | `u8` | `12` | none | Fast reconnect attempts for single-endpoint DC cases. | `me_one_retry = 6` |
|
||||
| `timeouts.me_one_timeout_ms` | `u64` | `1200` | none | Timeout per quick single-endpoint attempt (ms). | `me_one_timeout_ms = 1500` |
|
||||
|
||||
### 2) Pool size, keepalive, and reconnect policy
|
||||
|
||||
| Parameter | Type | Default | Constraints / validation | Runtime effect | Example |
|
||||
|---|---|---:|---|---|---|
|
||||
| `general.middle_proxy_pool_size` | `usize` | `8` | none | Target active ME writer pool size. | `middle_proxy_pool_size = 12` |
|
||||
| `general.middle_proxy_warm_standby` | `usize` | `16` | none | Reserved compatibility field in current revision (no active runtime consumer). | `middle_proxy_warm_standby = 16` |
|
||||
| `general.me_keepalive_enabled` | `bool` | `true` | none | Enables periodic ME keepalive/ping traffic. | `me_keepalive_enabled = true` |
|
||||
| `general.me_keepalive_interval_secs` | `u64` | `25` | none | Base keepalive interval (seconds). | `me_keepalive_interval_secs = 20` |
|
||||
| `general.me_keepalive_jitter_secs` | `u64` | `5` | none | Keepalive jitter to avoid synchronization bursts. | `me_keepalive_jitter_secs = 3` |
|
||||
| `general.me_keepalive_payload_random` | `bool` | `true` | none | Randomizes keepalive payload bytes. | `me_keepalive_payload_random = true` |
|
||||
| `general.me_warmup_stagger_enabled` | `bool` | `true` | none | Staggers extra ME warmup dials to avoid spikes. | `me_warmup_stagger_enabled = true` |
|
||||
| `general.me_warmup_step_delay_ms` | `u64` | `500` | none | Base delay between warmup dial steps (ms). | `me_warmup_step_delay_ms = 300` |
|
||||
| `general.me_warmup_step_jitter_ms` | `u64` | `300` | none | Additional random delay for warmup steps (ms). | `me_warmup_step_jitter_ms = 200` |
|
||||
| `general.me_reconnect_max_concurrent_per_dc` | `u32` | `8` | none | Limits concurrent reconnect workers per DC in health recovery. | `me_reconnect_max_concurrent_per_dc = 12` |
|
||||
| `general.me_reconnect_backoff_base_ms` | `u64` | `500` | none | Initial reconnect backoff (ms). | `me_reconnect_backoff_base_ms = 250` |
|
||||
| `general.me_reconnect_backoff_cap_ms` | `u64` | `30000` | none | Maximum reconnect backoff (ms). | `me_reconnect_backoff_cap_ms = 10000` |
|
||||
| `general.me_reconnect_fast_retry_count` | `u32` | `16` | none | Immediate retry budget before long backoff behavior. | `me_reconnect_fast_retry_count = 8` |
|
||||
|
||||
### 3) Reinit/hardswap, secret rotation, and degradation
|
||||
|
||||
| Parameter | Type | Default | Constraints / validation | Runtime effect | Example |
|
||||
|---|---|---:|---|---|---|
|
||||
| `general.hardswap` | `bool` | `true` | none | Enables generation-based ME hardswap strategy. | `hardswap = true` |
|
||||
| `general.me_reinit_every_secs` | `u64` | `900` | must be `> 0` | Periodic ME reinit interval. | `me_reinit_every_secs = 600` |
|
||||
| `general.me_hardswap_warmup_delay_min_ms` | `u64` | `1000` | must be `<= me_hardswap_warmup_delay_max_ms` | Lower bound for hardswap warmup dial spacing. | `me_hardswap_warmup_delay_min_ms = 500` |
|
||||
| `general.me_hardswap_warmup_delay_max_ms` | `u64` | `2000` | must be `> 0` | Upper bound for hardswap warmup dial spacing. | `me_hardswap_warmup_delay_max_ms = 1200` |
|
||||
| `general.me_hardswap_warmup_extra_passes` | `u8` | `3` | must be within `[0,10]` | Additional warmup passes after base pass. | `me_hardswap_warmup_extra_passes = 2` |
|
||||
| `general.me_hardswap_warmup_pass_backoff_base_ms` | `u64` | `500` | must be `> 0` | Base backoff between extra warmup passes. | `me_hardswap_warmup_pass_backoff_base_ms = 400` |
|
||||
| `general.me_config_stable_snapshots` | `u8` | `2` | must be `> 0` | Number of identical ME config snapshots required before apply. | `me_config_stable_snapshots = 3` |
|
||||
| `general.me_config_apply_cooldown_secs` | `u64` | `300` | none | Cooldown between applied ME map updates. | `me_config_apply_cooldown_secs = 120` |
|
||||
| `general.proxy_secret_stable_snapshots` | `u8` | `2` | must be `> 0` | Number of identical proxy-secret snapshots required before rotation. | `proxy_secret_stable_snapshots = 3` |
|
||||
| `general.proxy_secret_rotate_runtime` | `bool` | `true` | none | Enables runtime proxy-secret rotation. | `proxy_secret_rotate_runtime = true` |
|
||||
| `general.proxy_secret_len_max` | `usize` | `256` | must be within `[32,4096]` | Upper limit for accepted proxy-secret length. | `proxy_secret_len_max = 512` |
|
||||
| `general.update_every` | `Option<u64>` | `300` | if set: must be `> 0`; if `null`: legacy min fallback | Unified refresh interval for ME config + secret updater. | `update_every = 300` |
|
||||
| `general.me_pool_drain_ttl_secs` | `u64` | `90` | none | Time window where stale writers remain fallback-eligible. | `me_pool_drain_ttl_secs = 120` |
|
||||
| `general.me_pool_min_fresh_ratio` | `f32` | `0.8` | must be within `[0.0,1.0]` | Coverage threshold before stale generation can be drained. | `me_pool_min_fresh_ratio = 0.9` |
|
||||
| `general.me_reinit_drain_timeout_secs` | `u64` | `120` | `0` means no force-close; if `>0 && < TTL` it is bumped to TTL | Force-close timeout for draining stale writers. | `me_reinit_drain_timeout_secs = 0` |
|
||||
| `general.auto_degradation_enabled` | `bool` | `true` | none | Reserved compatibility flag in current revision (no active runtime consumer). | `auto_degradation_enabled = true` |
|
||||
| `general.degradation_min_unavailable_dc_groups` | `u8` | `2` | none | Reserved compatibility threshold in current revision (no active runtime consumer). | `degradation_min_unavailable_dc_groups = 2` |
|
||||
|
||||
## Deprecated / Legacy Parameters
|
||||
|
||||
| Parameter | Status | Replacement | Current behavior | Migration recommendation |
|
||||
|---|---|---|---|---|
|
||||
| `general.middle_proxy_nat_stun` | Deprecated | `network.stun_servers` | Merged into `network.stun_servers` only when `network.stun_servers` is not explicitly set. | Move value into `network.stun_servers` and remove legacy key. |
|
||||
| `general.middle_proxy_nat_stun_servers` | Deprecated | `network.stun_servers` | Merged into `network.stun_servers` only when `network.stun_servers` is not explicitly set. | Move values into `network.stun_servers` and remove legacy key. |
|
||||
| `general.proxy_secret_auto_reload_secs` | Deprecated | `general.update_every` | Used only when `update_every = null` (legacy fallback path). | Set `general.update_every` explicitly and remove legacy key. |
|
||||
| `general.proxy_config_auto_reload_secs` | Deprecated | `general.update_every` | Used only when `update_every = null` (legacy fallback path). | Set `general.update_every` explicitly and remove legacy key. |
|
||||
|
||||
## How Upstreams Are Configured
|
||||
|
||||
### Upstream schema
|
||||
|
||||
| Field | Applies to | Type | Required | Default | Meaning |
|
||||
|---|---|---|---|---|---|
|
||||
| `[[upstreams]].type` | all upstreams | `"direct" \| "socks4" \| "socks5"` | yes | n/a | Upstream transport type. |
|
||||
| `[[upstreams]].weight` | all upstreams | `u16` | no | `1` | Base weight for weighted-random selection. |
|
||||
| `[[upstreams]].enabled` | all upstreams | `bool` | no | `true` | Disabled entries are ignored at startup. |
|
||||
| `[[upstreams]].scopes` | all upstreams | `String` | no | `""` | Comma-separated scope tags for request-level routing. |
|
||||
| `interface` | `direct` | `Option<String>` | no | `null` | Interface name (e.g. `eth0`) or literal local IP for bind selection. |
|
||||
| `bind_addresses` | `direct` | `Option<Vec<IpAddr>>` | no | `null` | Explicit local source IP candidates (strict priority over `interface`). |
|
||||
| `address` | `socks4` | `String` | yes | n/a | SOCKS4 server endpoint (`ip:port` or `host:port`). |
|
||||
| `interface` | `socks4` | `Option<String>` | no | `null` | Used only for SOCKS server `ip:port` dial path. |
|
||||
| `user_id` | `socks4` | `Option<String>` | no | `null` | SOCKS4 user ID for CONNECT request. |
|
||||
| `address` | `socks5` | `String` | yes | n/a | SOCKS5 server endpoint (`ip:port` or `host:port`). |
|
||||
| `interface` | `socks5` | `Option<String>` | no | `null` | Used only for SOCKS server `ip:port` dial path. |
|
||||
| `username` | `socks5` | `Option<String>` | no | `null` | SOCKS5 username auth. |
|
||||
| `password` | `socks5` | `Option<String>` | no | `null` | SOCKS5 password auth. |
|
||||
|
||||
### Runtime rules (important)
|
||||
|
||||
1. If `[[upstreams]]` is omitted, loader injects one default `direct` upstream.
|
||||
2. Scope filtering is exact-token based:
|
||||
- when request scope is set -> only entries whose `scopes` contains that exact token;
|
||||
- when request scope is not set -> only entries with empty `scopes`.
|
||||
3. Healthy upstreams are selected by weighted random using: `weight * latency_factor`.
|
||||
4. If no healthy upstream exists in filtered set, random selection is used among filtered entries.
|
||||
5. `direct` bind resolution order:
|
||||
- `bind_addresses` candidates (same IP family as target) first;
|
||||
- if `interface` is an interface name and `bind_addresses` is set, each candidate IP is validated against addresses currently assigned to that interface;
|
||||
- invalid candidates are dropped with `WARN`;
|
||||
- if no valid candidate remains, connection falls back to unbound direct connect (`bind_ip=None`);
|
||||
- if no `bind_addresses` candidate, `interface` is used (literal IP or resolved interface primary IP).
|
||||
6. For `socks4/socks5` with `address` as hostname, interface binding is not supported and is ignored with warning.
|
||||
7. Runtime DNS overrides are used for upstream hostname resolution.
|
||||
8. In ME mode, the selected upstream is also used for ME TCP dial path.
|
||||
9. In ME mode for `direct` upstream with bind/interface, STUN reflection logic is bind-aware for KDF source material.
|
||||
10. In ME mode for SOCKS upstream, SOCKS `BND.ADDR/BND.PORT` is used for KDF when it is valid/public for the same family.
|
||||
|
||||
## Upstream Configuration Examples
|
||||
|
||||
### Example 1: Minimal direct upstream
|
||||
|
||||
```toml
|
||||
[[upstreams]]
|
||||
type = "direct"
|
||||
weight = 1
|
||||
enabled = true
|
||||
```
|
||||
|
||||
### Example 2: Direct with interface + explicit bind addresses
|
||||
|
||||
```toml
|
||||
[[upstreams]]
|
||||
type = "direct"
|
||||
interface = "eth0"
|
||||
bind_addresses = ["192.168.1.100", "192.168.1.101"]
|
||||
weight = 3
|
||||
enabled = true
|
||||
```
|
||||
|
||||
### Example 3: SOCKS5 upstream with authentication
|
||||
|
||||
```toml
|
||||
[[upstreams]]
|
||||
type = "socks5"
|
||||
address = "198.51.100.30:1080"
|
||||
username = "proxy-user"
|
||||
password = "proxy-pass"
|
||||
weight = 2
|
||||
enabled = true
|
||||
```
|
||||
|
||||
### Example 4: Mixed upstreams with scopes
|
||||
|
||||
```toml
|
||||
[[upstreams]]
|
||||
type = "direct"
|
||||
weight = 5
|
||||
enabled = true
|
||||
scopes = ""
|
||||
|
||||
[[upstreams]]
|
||||
type = "socks5"
|
||||
address = "203.0.113.40:1080"
|
||||
username = "edge"
|
||||
password = "edgepass"
|
||||
weight = 3
|
||||
enabled = true
|
||||
scopes = "premium,me"
|
||||
```
|
||||
|
||||
### Example 5: ME-focused tuning profile
|
||||
|
||||
```toml
|
||||
[general]
|
||||
use_middle_proxy = true
|
||||
proxy_secret_path = "proxy-secret"
|
||||
middle_proxy_nat_probe = true
|
||||
stun_nat_probe_concurrency = 16
|
||||
middle_proxy_pool_size = 12
|
||||
me_keepalive_enabled = true
|
||||
me_keepalive_interval_secs = 20
|
||||
me_keepalive_jitter_secs = 4
|
||||
me_reconnect_max_concurrent_per_dc = 12
|
||||
me_reconnect_backoff_base_ms = 300
|
||||
me_reconnect_backoff_cap_ms = 10000
|
||||
me_reconnect_fast_retry_count = 10
|
||||
hardswap = true
|
||||
me_reinit_every_secs = 600
|
||||
me_hardswap_warmup_delay_min_ms = 500
|
||||
me_hardswap_warmup_delay_max_ms = 1200
|
||||
me_hardswap_warmup_extra_passes = 2
|
||||
me_hardswap_warmup_pass_backoff_base_ms = 400
|
||||
me_config_stable_snapshots = 3
|
||||
me_config_apply_cooldown_secs = 120
|
||||
proxy_secret_stable_snapshots = 3
|
||||
proxy_secret_rotate_runtime = true
|
||||
proxy_secret_len_max = 512
|
||||
update_every = 300
|
||||
me_pool_drain_ttl_secs = 120
|
||||
me_pool_min_fresh_ratio = 0.9
|
||||
me_reinit_drain_timeout_secs = 180
|
||||
|
||||
[timeouts]
|
||||
me_one_retry = 8
|
||||
me_one_timeout_ms = 1200
|
||||
|
||||
[network]
|
||||
stun_use = true
|
||||
stun_tcp_fallback = true
|
||||
stun_servers = [
|
||||
"stun1.l.google.com:19302",
|
||||
"stun2.l.google.com:19302"
|
||||
]
|
||||
http_ip_detect_urls = [
|
||||
"https://api.ipify.org",
|
||||
"https://ifconfig.me/ip"
|
||||
]
|
||||
```
|
||||
219
docs/TUNING.ru.md
Normal file
219
docs/TUNING.ru.md
Normal file
@@ -0,0 +1,219 @@
|
||||
# Руководство по тюнингу Telemt: Middle-End и Upstreams
|
||||
|
||||
Документ описывает актуальное поведение Middle-End (ME) и маршрутизации через upstream на основе:
|
||||
- `src/config/types.rs`
|
||||
- `src/config/defaults.rs`
|
||||
- `src/config/load.rs`
|
||||
- `src/transport/upstream.rs`
|
||||
|
||||
Значения `Default` ниже — это значения из кода при отсутствии ключа в конфиге, а не обязательно значения из примеров `config.full.toml`.
|
||||
|
||||
## Параметры Middle-End
|
||||
|
||||
### 1) Базовый режим ME, NAT и STUN
|
||||
|
||||
| Параметр | Тип | Default | Ограничения / валидация | Влияние на runtime | Пример |
|
||||
|---|---|---:|---|---|---|
|
||||
| `general.use_middle_proxy` | `bool` | `true` | нет | Включает транспорт ME. При `false` используется Direct-режим. | `use_middle_proxy = true` |
|
||||
| `general.proxy_secret_path` | `Option<String>` | `"proxy-secret"` | путь может быть `null` | Путь к инфраструктурному proxy-secret Telegram. | `proxy_secret_path = "proxy-secret"` |
|
||||
| `general.middle_proxy_nat_ip` | `Option<IpAddr>` | `null` | валидный IP при задании | Ручной override публичного NAT IP для адресного материала ME. | `middle_proxy_nat_ip = "203.0.113.10"` |
|
||||
| `general.middle_proxy_nat_probe` | `bool` | `true` | авто-принудительно `true`, если `use_middle_proxy=true` | Включает NAT probing для ME. | `middle_proxy_nat_probe = true` |
|
||||
| `general.stun_nat_probe_concurrency` | `usize` | `8` | должно быть `> 0` | Максимум параллельных STUN-проб при NAT-детекте. | `stun_nat_probe_concurrency = 16` |
|
||||
| `network.stun_use` | `bool` | `true` | нет | Глобальный переключатель STUN. При `false` STUN отключен. | `stun_use = true` |
|
||||
| `network.stun_servers` | `Vec<String>` | встроенный публичный пул | удаляются дубликаты и пустые значения | Основной список STUN-серверов для NAT/public endpoint discovery. | `stun_servers = ["stun1.l.google.com:19302"]` |
|
||||
| `network.stun_tcp_fallback` | `bool` | `true` | нет | Включает TCP fallback, если UDP STUN недоступен. | `stun_tcp_fallback = true` |
|
||||
| `network.http_ip_detect_urls` | `Vec<String>` | `ifconfig.me` + `api.ipify.org` | нет | HTTP fallback для определения публичного IPv4 при недоступности STUN. | `http_ip_detect_urls = ["https://api.ipify.org"]` |
|
||||
| `general.stun_iface_mismatch_ignore` | `bool` | `false` | нет | Зарезервированный флаг в текущей ревизии (runtime его не использует). | `stun_iface_mismatch_ignore = false` |
|
||||
| `timeouts.me_one_retry` | `u8` | `12` | нет | Количество быстрых reconnect-попыток для DC с одним endpoint. | `me_one_retry = 6` |
|
||||
| `timeouts.me_one_timeout_ms` | `u64` | `1200` | нет | Таймаут одной быстрой попытки (мс). | `me_one_timeout_ms = 1500` |
|
||||
|
||||
### 2) Размер пула, keepalive и reconnect-политика
|
||||
|
||||
| Параметр | Тип | Default | Ограничения / валидация | Влияние на runtime | Пример |
|
||||
|---|---|---:|---|---|---|
|
||||
| `general.middle_proxy_pool_size` | `usize` | `8` | нет | Целевой размер активного пула ME-writer соединений. | `middle_proxy_pool_size = 12` |
|
||||
| `general.middle_proxy_warm_standby` | `usize` | `16` | нет | Зарезервированное поле совместимости в текущей ревизии (активного runtime-consumer нет). | `middle_proxy_warm_standby = 16` |
|
||||
| `general.me_keepalive_enabled` | `bool` | `true` | нет | Включает периодические keepalive/ping кадры ME. | `me_keepalive_enabled = true` |
|
||||
| `general.me_keepalive_interval_secs` | `u64` | `25` | нет | Базовый интервал keepalive (сек). | `me_keepalive_interval_secs = 20` |
|
||||
| `general.me_keepalive_jitter_secs` | `u64` | `5` | нет | Джиттер keepalive для предотвращения синхронных всплесков. | `me_keepalive_jitter_secs = 3` |
|
||||
| `general.me_keepalive_payload_random` | `bool` | `true` | нет | Рандомизирует payload keepalive-кадров. | `me_keepalive_payload_random = true` |
|
||||
| `general.me_warmup_stagger_enabled` | `bool` | `true` | нет | Включает staggered warmup дополнительных ME-коннектов. | `me_warmup_stagger_enabled = true` |
|
||||
| `general.me_warmup_step_delay_ms` | `u64` | `500` | нет | Базовая задержка между шагами warmup (мс). | `me_warmup_step_delay_ms = 300` |
|
||||
| `general.me_warmup_step_jitter_ms` | `u64` | `300` | нет | Дополнительный случайный warmup-джиттер (мс). | `me_warmup_step_jitter_ms = 200` |
|
||||
| `general.me_reconnect_max_concurrent_per_dc` | `u32` | `8` | нет | Ограничивает параллельные reconnect worker'ы на один DC. | `me_reconnect_max_concurrent_per_dc = 12` |
|
||||
| `general.me_reconnect_backoff_base_ms` | `u64` | `500` | нет | Начальный backoff reconnect (мс). | `me_reconnect_backoff_base_ms = 250` |
|
||||
| `general.me_reconnect_backoff_cap_ms` | `u64` | `30000` | нет | Верхняя граница backoff reconnect (мс). | `me_reconnect_backoff_cap_ms = 10000` |
|
||||
| `general.me_reconnect_fast_retry_count` | `u32` | `16` | нет | Бюджет быстрых retry до длинного backoff. | `me_reconnect_fast_retry_count = 8` |
|
||||
|
||||
### 3) Reinit/hardswap, ротация секрета и деградация
|
||||
|
||||
| Параметр | Тип | Default | Ограничения / валидация | Влияние на runtime | Пример |
|
||||
|---|---|---:|---|---|---|
|
||||
| `general.hardswap` | `bool` | `true` | нет | Включает generation-based стратегию hardswap для ME-пула. | `hardswap = true` |
|
||||
| `general.me_reinit_every_secs` | `u64` | `900` | должно быть `> 0` | Интервал периодического reinit ME-пула. | `me_reinit_every_secs = 600` |
|
||||
| `general.me_hardswap_warmup_delay_min_ms` | `u64` | `1000` | должно быть `<= me_hardswap_warmup_delay_max_ms` | Нижняя граница пауз между warmup dial попытками. | `me_hardswap_warmup_delay_min_ms = 500` |
|
||||
| `general.me_hardswap_warmup_delay_max_ms` | `u64` | `2000` | должно быть `> 0` | Верхняя граница пауз между warmup dial попытками. | `me_hardswap_warmup_delay_max_ms = 1200` |
|
||||
| `general.me_hardswap_warmup_extra_passes` | `u8` | `3` | диапазон `[0,10]` | Дополнительные warmup-проходы после базового. | `me_hardswap_warmup_extra_passes = 2` |
|
||||
| `general.me_hardswap_warmup_pass_backoff_base_ms` | `u64` | `500` | должно быть `> 0` | Базовый backoff между extra-pass в warmup. | `me_hardswap_warmup_pass_backoff_base_ms = 400` |
|
||||
| `general.me_config_stable_snapshots` | `u8` | `2` | должно быть `> 0` | Количество одинаковых snapshot перед применением ME map update. | `me_config_stable_snapshots = 3` |
|
||||
| `general.me_config_apply_cooldown_secs` | `u64` | `300` | нет | Cooldown между применёнными обновлениями ME map. | `me_config_apply_cooldown_secs = 120` |
|
||||
| `general.proxy_secret_stable_snapshots` | `u8` | `2` | должно быть `> 0` | Количество одинаковых snapshot перед runtime-rotation proxy-secret. | `proxy_secret_stable_snapshots = 3` |
|
||||
| `general.proxy_secret_rotate_runtime` | `bool` | `true` | нет | Включает runtime-ротацию proxy-secret. | `proxy_secret_rotate_runtime = true` |
|
||||
| `general.proxy_secret_len_max` | `usize` | `256` | диапазон `[32,4096]` | Верхний лимит длины принимаемого proxy-secret. | `proxy_secret_len_max = 512` |
|
||||
| `general.update_every` | `Option<u64>` | `300` | если задано: `> 0`; если `null`: fallback на legacy минимум | Единый интервал refresh для ME config + secret updater. | `update_every = 300` |
|
||||
| `general.me_pool_drain_ttl_secs` | `u64` | `90` | нет | Время, когда stale writer ещё может использоваться как fallback. | `me_pool_drain_ttl_secs = 120` |
|
||||
| `general.me_pool_min_fresh_ratio` | `f32` | `0.8` | диапазон `[0.0,1.0]` | Порог покрытия fresh-поколения перед drain старого поколения. | `me_pool_min_fresh_ratio = 0.9` |
|
||||
| `general.me_reinit_drain_timeout_secs` | `u64` | `120` | `0` = без force-close; если `>0 && < TTL`, поднимается до TTL | Таймаут force-close для draining stale writer. | `me_reinit_drain_timeout_secs = 0` |
|
||||
| `general.auto_degradation_enabled` | `bool` | `true` | нет | Зарезервированный флаг совместимости в текущей ревизии (активного runtime-consumer нет). | `auto_degradation_enabled = true` |
|
||||
| `general.degradation_min_unavailable_dc_groups` | `u8` | `2` | нет | Зарезервированный порог совместимости в текущей ревизии (активного runtime-consumer нет). | `degradation_min_unavailable_dc_groups = 2` |
|
||||
|
||||
## Устаревшие / legacy параметры
|
||||
|
||||
| Параметр | Статус | Замена | Текущее поведение | Рекомендация миграции |
|
||||
|---|---|---|---|---|
|
||||
| `general.middle_proxy_nat_stun` | Deprecated | `network.stun_servers` | Добавляется в `network.stun_servers`, только если `network.stun_servers` не задан явно. | Перенести значение в `network.stun_servers`, legacy-ключ удалить. |
|
||||
| `general.middle_proxy_nat_stun_servers` | Deprecated | `network.stun_servers` | Добавляется в `network.stun_servers`, только если `network.stun_servers` не задан явно. | Перенести значения в `network.stun_servers`, legacy-ключ удалить. |
|
||||
| `general.proxy_secret_auto_reload_secs` | Deprecated | `general.update_every` | Используется только если `update_every = null` (legacy fallback). | Явно задать `general.update_every`, legacy-ключ удалить. |
|
||||
| `general.proxy_config_auto_reload_secs` | Deprecated | `general.update_every` | Используется только если `update_every = null` (legacy fallback). | Явно задать `general.update_every`, legacy-ключ удалить. |
|
||||
|
||||
## Как конфигурируются Upstreams
|
||||
|
||||
### Схема upstream
|
||||
|
||||
| Поле | Применимость | Тип | Обязательно | Default | Назначение |
|
||||
|---|---|---|---|---|---|
|
||||
| `[[upstreams]].type` | все upstream | `"direct" \| "socks4" \| "socks5"` | да | n/a | Тип upstream транспорта. |
|
||||
| `[[upstreams]].weight` | все upstream | `u16` | нет | `1` | Базовый вес в weighted-random выборе. |
|
||||
| `[[upstreams]].enabled` | все upstream | `bool` | нет | `true` | Выключенные записи игнорируются на старте. |
|
||||
| `[[upstreams]].scopes` | все upstream | `String` | нет | `""` | Список scope-токенов через запятую для маршрутизации. |
|
||||
| `interface` | `direct` | `Option<String>` | нет | `null` | Имя интерфейса (например `eth0`) или literal локальный IP. |
|
||||
| `bind_addresses` | `direct` | `Option<Vec<IpAddr>>` | нет | `null` | Явные кандидаты source IP (имеют приоритет над `interface`). |
|
||||
| `address` | `socks4` | `String` | да | n/a | Адрес SOCKS4 сервера (`ip:port` или `host:port`). |
|
||||
| `interface` | `socks4` | `Option<String>` | нет | `null` | Используется только если `address` задан как `ip:port`. |
|
||||
| `user_id` | `socks4` | `Option<String>` | нет | `null` | SOCKS4 user ID в CONNECT-запросе. |
|
||||
| `address` | `socks5` | `String` | да | n/a | Адрес SOCKS5 сервера (`ip:port` или `host:port`). |
|
||||
| `interface` | `socks5` | `Option<String>` | нет | `null` | Используется только если `address` задан как `ip:port`. |
|
||||
| `username` | `socks5` | `Option<String>` | нет | `null` | Логин SOCKS5 auth. |
|
||||
| `password` | `socks5` | `Option<String>` | нет | `null` | Пароль SOCKS5 auth. |
|
||||
|
||||
### Runtime-правила
|
||||
|
||||
1. Если `[[upstreams]]` отсутствует, loader добавляет один upstream `direct` по умолчанию.
|
||||
2. Scope-фильтрация — по точному совпадению токена:
|
||||
- если scope запроса задан -> используются только записи, где `scopes` содержит такой же токен;
|
||||
- если scope запроса не задан -> используются только записи с пустым `scopes`.
|
||||
3. Среди healthy upstream используется weighted-random выбор: `weight * latency_factor`.
|
||||
4. Если в отфильтрованном наборе нет healthy upstream, выбирается случайный из отфильтрованных.
|
||||
5. Порядок выбора bind для `direct`:
|
||||
- сначала `bind_addresses` (только IP нужного семейства);
|
||||
- если одновременно заданы `interface` (имя) и `bind_addresses`, каждый IP проверяется на принадлежность интерфейсу;
|
||||
- несовпадающие IP отбрасываются с `WARN`;
|
||||
- если валидных IP не осталось, используется unbound direct connect (`bind_ip=None`);
|
||||
- если `bind_addresses` не подходит, применяется `interface` (literal IP или адрес интерфейса).
|
||||
6. Для `socks4/socks5` с `address` в виде hostname интерфейсный bind не поддерживается и игнорируется с предупреждением.
|
||||
7. Runtime DNS overrides применяются к резолвингу hostname в upstream-подключениях.
|
||||
8. В ME-режиме выбранный upstream также используется для ME TCP dial path.
|
||||
9. В ME-режиме для `direct` upstream с bind/interface STUN-рефлексия выполняется bind-aware для KDF материала.
|
||||
10. В ME-режиме для SOCKS upstream используются `BND.ADDR/BND.PORT` для KDF, если адрес валиден/публичен и соответствует IP family.
|
||||
|
||||
## Примеры конфигурации Upstreams
|
||||
|
||||
### Пример 1: минимальный direct upstream
|
||||
|
||||
```toml
|
||||
[[upstreams]]
|
||||
type = "direct"
|
||||
weight = 1
|
||||
enabled = true
|
||||
```
|
||||
|
||||
### Пример 2: direct с interface + явными bind IP
|
||||
|
||||
```toml
|
||||
[[upstreams]]
|
||||
type = "direct"
|
||||
interface = "eth0"
|
||||
bind_addresses = ["192.168.1.100", "192.168.1.101"]
|
||||
weight = 3
|
||||
enabled = true
|
||||
```
|
||||
|
||||
### Пример 3: SOCKS5 upstream с аутентификацией
|
||||
|
||||
```toml
|
||||
[[upstreams]]
|
||||
type = "socks5"
|
||||
address = "198.51.100.30:1080"
|
||||
username = "proxy-user"
|
||||
password = "proxy-pass"
|
||||
weight = 2
|
||||
enabled = true
|
||||
```
|
||||
|
||||
### Пример 4: смешанные upstream с scopes
|
||||
|
||||
```toml
|
||||
[[upstreams]]
|
||||
type = "direct"
|
||||
weight = 5
|
||||
enabled = true
|
||||
scopes = ""
|
||||
|
||||
[[upstreams]]
|
||||
type = "socks5"
|
||||
address = "203.0.113.40:1080"
|
||||
username = "edge"
|
||||
password = "edgepass"
|
||||
weight = 3
|
||||
enabled = true
|
||||
scopes = "premium,me"
|
||||
```
|
||||
|
||||
### Пример 5: профиль тюнинга под ME
|
||||
|
||||
```toml
|
||||
[general]
|
||||
use_middle_proxy = true
|
||||
proxy_secret_path = "proxy-secret"
|
||||
middle_proxy_nat_probe = true
|
||||
stun_nat_probe_concurrency = 16
|
||||
middle_proxy_pool_size = 12
|
||||
me_keepalive_enabled = true
|
||||
me_keepalive_interval_secs = 20
|
||||
me_keepalive_jitter_secs = 4
|
||||
me_reconnect_max_concurrent_per_dc = 12
|
||||
me_reconnect_backoff_base_ms = 300
|
||||
me_reconnect_backoff_cap_ms = 10000
|
||||
me_reconnect_fast_retry_count = 10
|
||||
hardswap = true
|
||||
me_reinit_every_secs = 600
|
||||
me_hardswap_warmup_delay_min_ms = 500
|
||||
me_hardswap_warmup_delay_max_ms = 1200
|
||||
me_hardswap_warmup_extra_passes = 2
|
||||
me_hardswap_warmup_pass_backoff_base_ms = 400
|
||||
me_config_stable_snapshots = 3
|
||||
me_config_apply_cooldown_secs = 120
|
||||
proxy_secret_stable_snapshots = 3
|
||||
proxy_secret_rotate_runtime = true
|
||||
proxy_secret_len_max = 512
|
||||
update_every = 300
|
||||
me_pool_drain_ttl_secs = 120
|
||||
me_pool_min_fresh_ratio = 0.9
|
||||
me_reinit_drain_timeout_secs = 180
|
||||
|
||||
[timeouts]
|
||||
me_one_retry = 8
|
||||
me_one_timeout_ms = 1200
|
||||
|
||||
[network]
|
||||
stun_use = true
|
||||
stun_tcp_fallback = true
|
||||
stun_servers = [
|
||||
"stun1.l.google.com:19302",
|
||||
"stun2.l.google.com:19302"
|
||||
]
|
||||
http_ip_detect_urls = [
|
||||
"https://api.ipify.org",
|
||||
"https://ifconfig.me/ip"
|
||||
]
|
||||
```
|
||||
@@ -3,6 +3,17 @@ use ipnetwork::IpNetwork;
|
||||
use serde::Deserialize;
|
||||
|
||||
// Helper defaults kept private to the config module.
|
||||
const DEFAULT_NETWORK_IPV6: Option<bool> = Some(false);
|
||||
const DEFAULT_STUN_TCP_FALLBACK: bool = true;
|
||||
const DEFAULT_MIDDLE_PROXY_WARM_STANDBY: usize = 16;
|
||||
const DEFAULT_ME_RECONNECT_MAX_CONCURRENT_PER_DC: u32 = 8;
|
||||
const DEFAULT_ME_RECONNECT_FAST_RETRY_COUNT: u32 = 16;
|
||||
const DEFAULT_UPSTREAM_CONNECT_RETRY_ATTEMPTS: u32 = 3;
|
||||
const DEFAULT_UPSTREAM_UNHEALTHY_FAIL_THRESHOLD: u32 = 4;
|
||||
const DEFAULT_LISTEN_ADDR_IPV6: &str = "::";
|
||||
const DEFAULT_ACCESS_USER: &str = "default";
|
||||
const DEFAULT_ACCESS_SECRET: &str = "00000000000000000000000000000000";
|
||||
|
||||
pub(crate) fn default_true() -> bool {
|
||||
true
|
||||
}
|
||||
@@ -12,7 +23,7 @@ pub(crate) fn default_port() -> u16 {
|
||||
}
|
||||
|
||||
pub(crate) fn default_tls_domain() -> String {
|
||||
"www.google.com".to_string()
|
||||
"petrovich.ru".to_string()
|
||||
}
|
||||
|
||||
pub(crate) fn default_mask_port() -> u16 {
|
||||
@@ -36,7 +47,7 @@ pub(crate) fn default_replay_window_secs() -> u64 {
|
||||
}
|
||||
|
||||
pub(crate) fn default_handshake_timeout() -> u64 {
|
||||
15
|
||||
30
|
||||
}
|
||||
|
||||
pub(crate) fn default_connect_timeout() -> u64 {
|
||||
@@ -51,17 +62,21 @@ pub(crate) fn default_ack_timeout() -> u64 {
|
||||
300
|
||||
}
|
||||
pub(crate) fn default_me_one_retry() -> u8 {
|
||||
3
|
||||
12
|
||||
}
|
||||
|
||||
pub(crate) fn default_me_one_timeout() -> u64 {
|
||||
1500
|
||||
1200
|
||||
}
|
||||
|
||||
pub(crate) fn default_listen_addr() -> String {
|
||||
"0.0.0.0".to_string()
|
||||
}
|
||||
|
||||
pub(crate) fn default_listen_addr_ipv4() -> Option<String> {
|
||||
Some(default_listen_addr())
|
||||
}
|
||||
|
||||
pub(crate) fn default_weight() -> u16 {
|
||||
1
|
||||
}
|
||||
@@ -77,6 +92,14 @@ pub(crate) fn default_prefer_4() -> u8 {
|
||||
4
|
||||
}
|
||||
|
||||
pub(crate) fn default_network_ipv6() -> Option<bool> {
|
||||
DEFAULT_NETWORK_IPV6
|
||||
}
|
||||
|
||||
pub(crate) fn default_stun_tcp_fallback() -> bool {
|
||||
DEFAULT_STUN_TCP_FALLBACK
|
||||
}
|
||||
|
||||
pub(crate) fn default_unknown_dc_log_path() -> Option<String> {
|
||||
Some("unknown-dc.txt".to_string())
|
||||
}
|
||||
@@ -85,6 +108,26 @@ pub(crate) fn default_pool_size() -> usize {
|
||||
8
|
||||
}
|
||||
|
||||
pub(crate) fn default_proxy_secret_path() -> Option<String> {
|
||||
Some("proxy-secret".to_string())
|
||||
}
|
||||
|
||||
pub(crate) fn default_middle_proxy_nat_stun() -> Option<String> {
|
||||
None
|
||||
}
|
||||
|
||||
pub(crate) fn default_middle_proxy_nat_stun_servers() -> Vec<String> {
|
||||
Vec::new()
|
||||
}
|
||||
|
||||
pub(crate) fn default_stun_nat_probe_concurrency() -> usize {
|
||||
8
|
||||
}
|
||||
|
||||
pub(crate) fn default_middle_proxy_warm_standby() -> usize {
|
||||
DEFAULT_MIDDLE_PROXY_WARM_STANDBY
|
||||
}
|
||||
|
||||
pub(crate) fn default_keepalive_interval() -> u64 {
|
||||
25
|
||||
}
|
||||
@@ -109,6 +152,26 @@ pub(crate) fn default_reconnect_backoff_cap_ms() -> u64 {
|
||||
30_000
|
||||
}
|
||||
|
||||
pub(crate) fn default_me_reconnect_max_concurrent_per_dc() -> u32 {
|
||||
DEFAULT_ME_RECONNECT_MAX_CONCURRENT_PER_DC
|
||||
}
|
||||
|
||||
pub(crate) fn default_me_reconnect_fast_retry_count() -> u32 {
|
||||
DEFAULT_ME_RECONNECT_FAST_RETRY_COUNT
|
||||
}
|
||||
|
||||
pub(crate) fn default_upstream_connect_retry_attempts() -> u32 {
|
||||
DEFAULT_UPSTREAM_CONNECT_RETRY_ATTEMPTS
|
||||
}
|
||||
|
||||
pub(crate) fn default_upstream_connect_retry_backoff_ms() -> u64 {
|
||||
250
|
||||
}
|
||||
|
||||
pub(crate) fn default_upstream_unhealthy_fail_threshold() -> u32 {
|
||||
DEFAULT_UPSTREAM_UNHEALTHY_FAIL_THRESHOLD
|
||||
}
|
||||
|
||||
pub(crate) fn default_crypto_pending_buffer() -> usize {
|
||||
256 * 1024
|
||||
}
|
||||
@@ -121,6 +184,30 @@ pub(crate) fn default_desync_all_full() -> bool {
|
||||
false
|
||||
}
|
||||
|
||||
pub(crate) fn default_me_route_backpressure_base_timeout_ms() -> u64 {
|
||||
25
|
||||
}
|
||||
|
||||
pub(crate) fn default_me_route_backpressure_high_timeout_ms() -> u64 {
|
||||
120
|
||||
}
|
||||
|
||||
pub(crate) fn default_me_route_backpressure_high_watermark_pct() -> u8 {
|
||||
80
|
||||
}
|
||||
|
||||
pub(crate) fn default_beobachten_minutes() -> u64 {
|
||||
10
|
||||
}
|
||||
|
||||
pub(crate) fn default_beobachten_flush_secs() -> u64 {
|
||||
15
|
||||
}
|
||||
|
||||
pub(crate) fn default_beobachten_file() -> String {
|
||||
"cache/beobachten.txt".to_string()
|
||||
}
|
||||
|
||||
pub(crate) fn default_tls_new_session_tickets() -> u8 {
|
||||
0
|
||||
}
|
||||
@@ -171,15 +258,59 @@ pub(crate) fn default_cache_public_ip_path() -> String {
|
||||
}
|
||||
|
||||
pub(crate) fn default_proxy_secret_reload_secs() -> u64 {
|
||||
1 * 60 * 60
|
||||
60 * 60
|
||||
}
|
||||
|
||||
pub(crate) fn default_proxy_config_reload_secs() -> u64 {
|
||||
1 * 60 * 60
|
||||
60 * 60
|
||||
}
|
||||
|
||||
pub(crate) fn default_update_every_secs() -> u64 {
|
||||
1 * 30 * 60
|
||||
5 * 60
|
||||
}
|
||||
|
||||
pub(crate) fn default_update_every() -> Option<u64> {
|
||||
Some(default_update_every_secs())
|
||||
}
|
||||
|
||||
pub(crate) fn default_me_reinit_every_secs() -> u64 {
|
||||
15 * 60
|
||||
}
|
||||
|
||||
pub(crate) fn default_me_hardswap_warmup_delay_min_ms() -> u64 {
|
||||
1000
|
||||
}
|
||||
|
||||
pub(crate) fn default_me_hardswap_warmup_delay_max_ms() -> u64 {
|
||||
2000
|
||||
}
|
||||
|
||||
pub(crate) fn default_me_hardswap_warmup_extra_passes() -> u8 {
|
||||
3
|
||||
}
|
||||
|
||||
pub(crate) fn default_me_hardswap_warmup_pass_backoff_base_ms() -> u64 {
|
||||
500
|
||||
}
|
||||
|
||||
pub(crate) fn default_me_config_stable_snapshots() -> u8 {
|
||||
2
|
||||
}
|
||||
|
||||
pub(crate) fn default_me_config_apply_cooldown_secs() -> u64 {
|
||||
300
|
||||
}
|
||||
|
||||
pub(crate) fn default_proxy_secret_stable_snapshots() -> u8 {
|
||||
2
|
||||
}
|
||||
|
||||
pub(crate) fn default_proxy_secret_rotate_runtime() -> bool {
|
||||
true
|
||||
}
|
||||
|
||||
pub(crate) fn default_proxy_secret_len_max() -> usize {
|
||||
256
|
||||
}
|
||||
|
||||
pub(crate) fn default_me_reinit_drain_timeout_secs() -> u64 {
|
||||
@@ -214,6 +345,21 @@ pub(crate) fn default_degradation_min_unavailable_dc_groups() -> u8 {
|
||||
2
|
||||
}
|
||||
|
||||
pub(crate) fn default_listen_addr_ipv6() -> String {
|
||||
DEFAULT_LISTEN_ADDR_IPV6.to_string()
|
||||
}
|
||||
|
||||
pub(crate) fn default_listen_addr_ipv6_opt() -> Option<String> {
|
||||
Some(default_listen_addr_ipv6())
|
||||
}
|
||||
|
||||
pub(crate) fn default_access_users() -> HashMap<String, String> {
|
||||
HashMap::from([(
|
||||
DEFAULT_ACCESS_USER.to_string(),
|
||||
DEFAULT_ACCESS_SECRET.to_string(),
|
||||
)])
|
||||
}
|
||||
|
||||
// Custom deserializer helpers
|
||||
|
||||
#[derive(Deserialize)]
|
||||
|
||||
@@ -16,6 +16,8 @@
|
||||
//! | `general` | `me_pool_drain_ttl_secs` | Applied on next ME map update |
|
||||
//! | `general` | `me_pool_min_fresh_ratio` | Applied on next ME map update |
|
||||
//! | `general` | `me_reinit_drain_timeout_secs`| Applied on next ME map update |
|
||||
//! | `general` | `telemetry` / `me_*_policy` | Applied immediately |
|
||||
//! | `network` | `dns_overrides` | Applied immediately |
|
||||
//! | `access` | All user/quota fields | Effective immediately |
|
||||
//!
|
||||
//! Fields that require re-binding sockets (`server.port`, `censorship.*`,
|
||||
@@ -29,7 +31,7 @@ use notify::{EventKind, RecursiveMode, Watcher, recommended_watcher};
|
||||
use tokio::sync::{mpsc, watch};
|
||||
use tracing::{error, info, warn};
|
||||
|
||||
use crate::config::LogLevel;
|
||||
use crate::config::{LogLevel, MeSocksKdfPolicy, MeTelemetryLevel};
|
||||
use super::load::ProxyConfig;
|
||||
|
||||
// ── Hot fields ────────────────────────────────────────────────────────────────
|
||||
@@ -39,6 +41,7 @@ use super::load::ProxyConfig;
|
||||
pub struct HotFields {
|
||||
pub log_level: LogLevel,
|
||||
pub ad_tag: Option<String>,
|
||||
pub dns_overrides: Vec<String>,
|
||||
pub middle_proxy_pool_size: usize,
|
||||
pub desync_all_full: bool,
|
||||
pub update_every_secs: u64,
|
||||
@@ -50,6 +53,13 @@ pub struct HotFields {
|
||||
pub me_keepalive_interval_secs: u64,
|
||||
pub me_keepalive_jitter_secs: u64,
|
||||
pub me_keepalive_payload_random: bool,
|
||||
pub telemetry_core_enabled: bool,
|
||||
pub telemetry_user_enabled: bool,
|
||||
pub telemetry_me_level: MeTelemetryLevel,
|
||||
pub me_socks_kdf_policy: MeSocksKdfPolicy,
|
||||
pub me_route_backpressure_base_timeout_ms: u64,
|
||||
pub me_route_backpressure_high_timeout_ms: u64,
|
||||
pub me_route_backpressure_high_watermark_pct: u8,
|
||||
pub access: crate::config::AccessConfig,
|
||||
}
|
||||
|
||||
@@ -58,6 +68,7 @@ impl HotFields {
|
||||
Self {
|
||||
log_level: cfg.general.log_level.clone(),
|
||||
ad_tag: cfg.general.ad_tag.clone(),
|
||||
dns_overrides: cfg.network.dns_overrides.clone(),
|
||||
middle_proxy_pool_size: cfg.general.middle_proxy_pool_size,
|
||||
desync_all_full: cfg.general.desync_all_full,
|
||||
update_every_secs: cfg.general.effective_update_every_secs(),
|
||||
@@ -69,6 +80,13 @@ impl HotFields {
|
||||
me_keepalive_interval_secs: cfg.general.me_keepalive_interval_secs,
|
||||
me_keepalive_jitter_secs: cfg.general.me_keepalive_jitter_secs,
|
||||
me_keepalive_payload_random: cfg.general.me_keepalive_payload_random,
|
||||
telemetry_core_enabled: cfg.general.telemetry.core_enabled,
|
||||
telemetry_user_enabled: cfg.general.telemetry.user_enabled,
|
||||
telemetry_me_level: cfg.general.telemetry.me_level,
|
||||
me_socks_kdf_policy: cfg.general.me_socks_kdf_policy,
|
||||
me_route_backpressure_base_timeout_ms: cfg.general.me_route_backpressure_base_timeout_ms,
|
||||
me_route_backpressure_high_timeout_ms: cfg.general.me_route_backpressure_high_timeout_ms,
|
||||
me_route_backpressure_high_watermark_pct: cfg.general.me_route_backpressure_high_watermark_pct,
|
||||
access: cfg.access.clone(),
|
||||
}
|
||||
}
|
||||
@@ -96,6 +114,17 @@ fn warn_non_hot_changes(old: &ProxyConfig, new: &ProxyConfig) {
|
||||
if old.general.use_middle_proxy != new.general.use_middle_proxy {
|
||||
warn!("config reload: use_middle_proxy changed; restart required");
|
||||
}
|
||||
if old.general.stun_nat_probe_concurrency != new.general.stun_nat_probe_concurrency {
|
||||
warn!("config reload: general.stun_nat_probe_concurrency changed; restart required");
|
||||
}
|
||||
if old.general.upstream_connect_retry_attempts != new.general.upstream_connect_retry_attempts
|
||||
|| old.general.upstream_connect_retry_backoff_ms
|
||||
!= new.general.upstream_connect_retry_backoff_ms
|
||||
|| old.general.upstream_unhealthy_fail_threshold
|
||||
!= new.general.upstream_unhealthy_fail_threshold
|
||||
{
|
||||
warn!("config reload: general.upstream_* changed; restart required");
|
||||
}
|
||||
}
|
||||
|
||||
/// Resolve the public host for link generation — mirrors the logic in main.rs.
|
||||
@@ -186,6 +215,13 @@ fn log_changes(
|
||||
);
|
||||
}
|
||||
|
||||
if old_hot.dns_overrides != new_hot.dns_overrides {
|
||||
info!(
|
||||
"config reload: network.dns_overrides updated ({} entries)",
|
||||
new_hot.dns_overrides.len()
|
||||
);
|
||||
}
|
||||
|
||||
if old_hot.middle_proxy_pool_size != new_hot.middle_proxy_pool_size {
|
||||
info!(
|
||||
"config reload: middle_proxy_pool_size: {} → {}",
|
||||
@@ -249,6 +285,41 @@ fn log_changes(
|
||||
);
|
||||
}
|
||||
|
||||
if old_hot.telemetry_core_enabled != new_hot.telemetry_core_enabled
|
||||
|| old_hot.telemetry_user_enabled != new_hot.telemetry_user_enabled
|
||||
|| old_hot.telemetry_me_level != new_hot.telemetry_me_level
|
||||
{
|
||||
info!(
|
||||
"config reload: telemetry: core_enabled={} user_enabled={} me_level={}",
|
||||
new_hot.telemetry_core_enabled,
|
||||
new_hot.telemetry_user_enabled,
|
||||
new_hot.telemetry_me_level,
|
||||
);
|
||||
}
|
||||
|
||||
if old_hot.me_socks_kdf_policy != new_hot.me_socks_kdf_policy {
|
||||
info!(
|
||||
"config reload: me_socks_kdf_policy: {:?} → {:?}",
|
||||
old_hot.me_socks_kdf_policy,
|
||||
new_hot.me_socks_kdf_policy,
|
||||
);
|
||||
}
|
||||
|
||||
if old_hot.me_route_backpressure_base_timeout_ms
|
||||
!= new_hot.me_route_backpressure_base_timeout_ms
|
||||
|| old_hot.me_route_backpressure_high_timeout_ms
|
||||
!= new_hot.me_route_backpressure_high_timeout_ms
|
||||
|| old_hot.me_route_backpressure_high_watermark_pct
|
||||
!= new_hot.me_route_backpressure_high_watermark_pct
|
||||
{
|
||||
info!(
|
||||
"config reload: me_route_backpressure: base={}ms high={}ms watermark={}%",
|
||||
new_hot.me_route_backpressure_base_timeout_ms,
|
||||
new_hot.me_route_backpressure_high_timeout_ms,
|
||||
new_hot.me_route_backpressure_high_watermark_pct,
|
||||
);
|
||||
}
|
||||
|
||||
if old_hot.access.users != new_hot.access.users {
|
||||
let mut added: Vec<&String> = new_hot.access.users.keys()
|
||||
.filter(|u| !old_hot.access.users.contains_key(*u))
|
||||
@@ -351,6 +422,16 @@ fn reload_config(
|
||||
return;
|
||||
}
|
||||
|
||||
if old_hot.dns_overrides != new_hot.dns_overrides
|
||||
&& let Err(e) = crate::network::dns_overrides::install_entries(&new_hot.dns_overrides)
|
||||
{
|
||||
error!(
|
||||
"config reload: invalid network.dns_overrides: {}; keeping old config",
|
||||
e
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
warn_non_hot_changes(&old_cfg, &new_cfg);
|
||||
log_changes(&old_hot, &new_hot, &new_cfg, log_tx, detected_ip_v4, detected_ip_v6);
|
||||
config_tx.send(Arc::new(new_cfg)).ok();
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
#![allow(deprecated)]
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::net::IpAddr;
|
||||
use std::path::Path;
|
||||
@@ -63,6 +65,33 @@ fn validate_network_cfg(net: &mut NetworkConfig) -> Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn push_unique_nonempty(target: &mut Vec<String>, value: String) {
|
||||
let trimmed = value.trim();
|
||||
if trimmed.is_empty() {
|
||||
return;
|
||||
}
|
||||
if !target.iter().any(|existing| existing == trimmed) {
|
||||
target.push(trimmed.to_string());
|
||||
}
|
||||
}
|
||||
|
||||
fn is_valid_ad_tag(tag: &str) -> bool {
|
||||
tag.len() == 32 && tag.chars().all(|ch| ch.is_ascii_hexdigit())
|
||||
}
|
||||
|
||||
fn sanitize_ad_tag(ad_tag: &mut Option<String>) {
|
||||
let Some(tag) = ad_tag.as_ref() else {
|
||||
return;
|
||||
};
|
||||
|
||||
if !is_valid_ad_tag(tag) {
|
||||
warn!(
|
||||
"Invalid general.ad_tag value, expected exactly 32 hex chars; ad_tag is disabled"
|
||||
);
|
||||
*ad_tag = None;
|
||||
}
|
||||
}
|
||||
|
||||
// ============= Main Config =============
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||
@@ -114,8 +143,65 @@ impl ProxyConfig {
|
||||
let base_dir = path.as_ref().parent().unwrap_or(Path::new("."));
|
||||
let processed = preprocess_includes(&content, base_dir, 0)?;
|
||||
|
||||
let mut config: ProxyConfig =
|
||||
let parsed_toml: toml::Value =
|
||||
toml::from_str(&processed).map_err(|e| ProxyError::Config(e.to_string()))?;
|
||||
let general_table = parsed_toml
|
||||
.get("general")
|
||||
.and_then(|value| value.as_table());
|
||||
let network_table = parsed_toml
|
||||
.get("network")
|
||||
.and_then(|value| value.as_table());
|
||||
let update_every_is_explicit = general_table
|
||||
.map(|table| table.contains_key("update_every"))
|
||||
.unwrap_or(false);
|
||||
let legacy_secret_is_explicit = general_table
|
||||
.map(|table| table.contains_key("proxy_secret_auto_reload_secs"))
|
||||
.unwrap_or(false);
|
||||
let legacy_config_is_explicit = general_table
|
||||
.map(|table| table.contains_key("proxy_config_auto_reload_secs"))
|
||||
.unwrap_or(false);
|
||||
let stun_servers_is_explicit = network_table
|
||||
.map(|table| table.contains_key("stun_servers"))
|
||||
.unwrap_or(false);
|
||||
|
||||
let mut config: ProxyConfig =
|
||||
parsed_toml.try_into().map_err(|e| ProxyError::Config(e.to_string()))?;
|
||||
|
||||
if !update_every_is_explicit && (legacy_secret_is_explicit || legacy_config_is_explicit) {
|
||||
config.general.update_every = None;
|
||||
}
|
||||
|
||||
let legacy_nat_stun = config.general.middle_proxy_nat_stun.take();
|
||||
let legacy_nat_stun_servers = std::mem::take(&mut config.general.middle_proxy_nat_stun_servers);
|
||||
let legacy_nat_stun_used = legacy_nat_stun.is_some() || !legacy_nat_stun_servers.is_empty();
|
||||
if stun_servers_is_explicit {
|
||||
let mut explicit_stun_servers = Vec::new();
|
||||
for stun in std::mem::take(&mut config.network.stun_servers) {
|
||||
push_unique_nonempty(&mut explicit_stun_servers, stun);
|
||||
}
|
||||
config.network.stun_servers = explicit_stun_servers;
|
||||
|
||||
if legacy_nat_stun_used {
|
||||
warn!("general.middle_proxy_nat_stun and general.middle_proxy_nat_stun_servers are ignored because network.stun_servers is explicitly set");
|
||||
}
|
||||
} else {
|
||||
// Keep the default STUN pool unless network.stun_servers is explicitly overridden.
|
||||
let mut unified_stun_servers = default_stun_servers();
|
||||
if let Some(stun) = legacy_nat_stun {
|
||||
push_unique_nonempty(&mut unified_stun_servers, stun);
|
||||
}
|
||||
for stun in legacy_nat_stun_servers {
|
||||
push_unique_nonempty(&mut unified_stun_servers, stun);
|
||||
}
|
||||
|
||||
config.network.stun_servers = unified_stun_servers;
|
||||
|
||||
if legacy_nat_stun_used {
|
||||
warn!("general.middle_proxy_nat_stun and general.middle_proxy_nat_stun_servers are deprecated; use network.stun_servers");
|
||||
}
|
||||
}
|
||||
|
||||
sanitize_ad_tag(&mut config.general.ad_tag);
|
||||
|
||||
if let Some(update_every) = config.general.update_every {
|
||||
if update_every == 0 {
|
||||
@@ -145,12 +231,118 @@ impl ProxyConfig {
|
||||
}
|
||||
}
|
||||
|
||||
if config.general.stun_nat_probe_concurrency == 0 {
|
||||
return Err(ProxyError::Config(
|
||||
"general.stun_nat_probe_concurrency must be > 0".to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
if config.general.upstream_connect_retry_attempts == 0 {
|
||||
return Err(ProxyError::Config(
|
||||
"general.upstream_connect_retry_attempts must be > 0".to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
if config.general.upstream_unhealthy_fail_threshold == 0 {
|
||||
return Err(ProxyError::Config(
|
||||
"general.upstream_unhealthy_fail_threshold must be > 0".to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
if config.general.me_reinit_every_secs == 0 {
|
||||
return Err(ProxyError::Config(
|
||||
"general.me_reinit_every_secs must be > 0".to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
if config.general.beobachten_minutes == 0 {
|
||||
return Err(ProxyError::Config(
|
||||
"general.beobachten_minutes must be > 0".to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
if config.general.beobachten_flush_secs == 0 {
|
||||
return Err(ProxyError::Config(
|
||||
"general.beobachten_flush_secs must be > 0".to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
if config.general.beobachten_file.trim().is_empty() {
|
||||
return Err(ProxyError::Config(
|
||||
"general.beobachten_file cannot be empty".to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
if config.general.me_hardswap_warmup_delay_max_ms == 0 {
|
||||
return Err(ProxyError::Config(
|
||||
"general.me_hardswap_warmup_delay_max_ms must be > 0".to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
if config.general.me_hardswap_warmup_delay_min_ms
|
||||
> config.general.me_hardswap_warmup_delay_max_ms
|
||||
{
|
||||
return Err(ProxyError::Config(
|
||||
"general.me_hardswap_warmup_delay_min_ms must be <= general.me_hardswap_warmup_delay_max_ms".to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
if config.general.me_hardswap_warmup_extra_passes > 10 {
|
||||
return Err(ProxyError::Config(
|
||||
"general.me_hardswap_warmup_extra_passes must be within [0, 10]".to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
if config.general.me_hardswap_warmup_pass_backoff_base_ms == 0 {
|
||||
return Err(ProxyError::Config(
|
||||
"general.me_hardswap_warmup_pass_backoff_base_ms must be > 0".to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
if config.general.me_config_stable_snapshots == 0 {
|
||||
return Err(ProxyError::Config(
|
||||
"general.me_config_stable_snapshots must be > 0".to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
if config.general.proxy_secret_stable_snapshots == 0 {
|
||||
return Err(ProxyError::Config(
|
||||
"general.proxy_secret_stable_snapshots must be > 0".to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
if !(32..=4096).contains(&config.general.proxy_secret_len_max) {
|
||||
return Err(ProxyError::Config(
|
||||
"general.proxy_secret_len_max must be within [32, 4096]".to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
if !(0.0..=1.0).contains(&config.general.me_pool_min_fresh_ratio) {
|
||||
return Err(ProxyError::Config(
|
||||
"general.me_pool_min_fresh_ratio must be within [0.0, 1.0]".to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
if config.general.me_route_backpressure_base_timeout_ms == 0 {
|
||||
return Err(ProxyError::Config(
|
||||
"general.me_route_backpressure_base_timeout_ms must be > 0".to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
if config.general.me_route_backpressure_high_timeout_ms
|
||||
< config.general.me_route_backpressure_base_timeout_ms
|
||||
{
|
||||
return Err(ProxyError::Config(
|
||||
"general.me_route_backpressure_high_timeout_ms must be >= general.me_route_backpressure_base_timeout_ms".to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
if !(1..=100).contains(&config.general.me_route_backpressure_high_watermark_pct) {
|
||||
return Err(ProxyError::Config(
|
||||
"general.me_route_backpressure_high_watermark_pct must be within [1, 100]".to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
if config.general.effective_me_pool_force_close_secs() > 0
|
||||
&& config.general.effective_me_pool_force_close_secs()
|
||||
< config.general.me_pool_drain_ttl_secs
|
||||
@@ -239,6 +431,7 @@ impl ProxyConfig {
|
||||
}
|
||||
|
||||
validate_network_cfg(&mut config.network)?;
|
||||
crate::network::dns_overrides::validate_entries(&config.network.dns_overrides)?;
|
||||
|
||||
if config.general.use_middle_proxy && config.network.ipv6 == Some(true) {
|
||||
warn!("IPv6 with Middle Proxy is experimental and may cause KDF address mismatch; consider disabling IPv6 or ME");
|
||||
@@ -276,23 +469,25 @@ impl ProxyConfig {
|
||||
reuse_allow: false,
|
||||
});
|
||||
}
|
||||
if let Some(ipv6_str) = &config.server.listen_addr_ipv6 {
|
||||
if let Ok(ipv6) = ipv6_str.parse::<IpAddr>() {
|
||||
config.server.listeners.push(ListenerConfig {
|
||||
ip: ipv6,
|
||||
announce: None,
|
||||
announce_ip: None,
|
||||
proxy_protocol: None,
|
||||
reuse_allow: false,
|
||||
});
|
||||
}
|
||||
if let Some(ipv6_str) = &config.server.listen_addr_ipv6
|
||||
&& let Ok(ipv6) = ipv6_str.parse::<IpAddr>()
|
||||
{
|
||||
config.server.listeners.push(ListenerConfig {
|
||||
ip: ipv6,
|
||||
announce: None,
|
||||
announce_ip: None,
|
||||
proxy_protocol: None,
|
||||
reuse_allow: false,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Migration: announce_ip → announce for each listener.
|
||||
for listener in &mut config.server.listeners {
|
||||
if listener.announce.is_none() && listener.announce_ip.is_some() {
|
||||
listener.announce = Some(listener.announce_ip.unwrap().to_string());
|
||||
if listener.announce.is_none()
|
||||
&& let Some(ip) = listener.announce_ip.take()
|
||||
{
|
||||
listener.announce = Some(ip.to_string());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -339,14 +534,18 @@ impl ProxyConfig {
|
||||
|
||||
if let Some(tag) = &self.general.ad_tag {
|
||||
let zeros = "00000000000000000000000000000000";
|
||||
if !is_valid_ad_tag(tag) {
|
||||
return Err(ProxyError::Config(
|
||||
"general.ad_tag must be exactly 32 hex characters".to_string(),
|
||||
));
|
||||
}
|
||||
if tag == zeros {
|
||||
warn!("ad_tag is all zeros; register a valid proxy tag via @MTProxybot to enable sponsored channel");
|
||||
}
|
||||
if tag.len() != 32 || tag.chars().any(|c| !c.is_ascii_hexdigit()) {
|
||||
warn!("ad_tag is not a 32-char hex string; ensure you use value issued by @MTProxybot");
|
||||
}
|
||||
}
|
||||
|
||||
crate::network::dns_overrides::validate_entries(&self.network.dns_overrides)?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
@@ -355,6 +554,90 @@ impl ProxyConfig {
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn serde_defaults_remain_unchanged_for_present_sections() {
|
||||
let toml = r#"
|
||||
[network]
|
||||
[general]
|
||||
[server]
|
||||
[access]
|
||||
"#;
|
||||
let cfg: ProxyConfig = toml::from_str(toml).unwrap();
|
||||
|
||||
assert_eq!(cfg.network.ipv6, default_network_ipv6());
|
||||
assert_eq!(cfg.network.stun_use, default_true());
|
||||
assert_eq!(cfg.network.stun_tcp_fallback, default_stun_tcp_fallback());
|
||||
assert_eq!(
|
||||
cfg.general.middle_proxy_warm_standby,
|
||||
default_middle_proxy_warm_standby()
|
||||
);
|
||||
assert_eq!(
|
||||
cfg.general.me_reconnect_max_concurrent_per_dc,
|
||||
default_me_reconnect_max_concurrent_per_dc()
|
||||
);
|
||||
assert_eq!(
|
||||
cfg.general.me_reconnect_fast_retry_count,
|
||||
default_me_reconnect_fast_retry_count()
|
||||
);
|
||||
assert_eq!(
|
||||
cfg.general.upstream_connect_retry_attempts,
|
||||
default_upstream_connect_retry_attempts()
|
||||
);
|
||||
assert_eq!(
|
||||
cfg.general.upstream_connect_retry_backoff_ms,
|
||||
default_upstream_connect_retry_backoff_ms()
|
||||
);
|
||||
assert_eq!(
|
||||
cfg.general.upstream_unhealthy_fail_threshold,
|
||||
default_upstream_unhealthy_fail_threshold()
|
||||
);
|
||||
assert_eq!(cfg.general.update_every, default_update_every());
|
||||
assert_eq!(cfg.server.listen_addr_ipv4, default_listen_addr_ipv4());
|
||||
assert_eq!(cfg.server.listen_addr_ipv6, default_listen_addr_ipv6_opt());
|
||||
assert_eq!(cfg.access.users, default_access_users());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn impl_defaults_are_sourced_from_default_helpers() {
|
||||
let network = NetworkConfig::default();
|
||||
assert_eq!(network.ipv6, default_network_ipv6());
|
||||
assert_eq!(network.stun_use, default_true());
|
||||
assert_eq!(network.stun_tcp_fallback, default_stun_tcp_fallback());
|
||||
|
||||
let general = GeneralConfig::default();
|
||||
assert_eq!(
|
||||
general.middle_proxy_warm_standby,
|
||||
default_middle_proxy_warm_standby()
|
||||
);
|
||||
assert_eq!(
|
||||
general.me_reconnect_max_concurrent_per_dc,
|
||||
default_me_reconnect_max_concurrent_per_dc()
|
||||
);
|
||||
assert_eq!(
|
||||
general.me_reconnect_fast_retry_count,
|
||||
default_me_reconnect_fast_retry_count()
|
||||
);
|
||||
assert_eq!(
|
||||
general.upstream_connect_retry_attempts,
|
||||
default_upstream_connect_retry_attempts()
|
||||
);
|
||||
assert_eq!(
|
||||
general.upstream_connect_retry_backoff_ms,
|
||||
default_upstream_connect_retry_backoff_ms()
|
||||
);
|
||||
assert_eq!(
|
||||
general.upstream_unhealthy_fail_threshold,
|
||||
default_upstream_unhealthy_fail_threshold()
|
||||
);
|
||||
assert_eq!(general.update_every, default_update_every());
|
||||
|
||||
let server = ServerConfig::default();
|
||||
assert_eq!(server.listen_addr_ipv6, Some(default_listen_addr_ipv6()));
|
||||
|
||||
let access = AccessConfig::default();
|
||||
assert_eq!(access.users, default_access_users());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn dc_overrides_allow_string_and_array() {
|
||||
let toml = r#"
|
||||
@@ -458,6 +741,281 @@ mod tests {
|
||||
let _ = std::fs::remove_file(path);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn stun_nat_probe_concurrency_zero_is_rejected() {
|
||||
let toml = r#"
|
||||
[general]
|
||||
stun_nat_probe_concurrency = 0
|
||||
|
||||
[censorship]
|
||||
tls_domain = "example.com"
|
||||
|
||||
[access.users]
|
||||
user = "00000000000000000000000000000000"
|
||||
"#;
|
||||
let dir = std::env::temp_dir();
|
||||
let path = dir.join("telemt_stun_nat_probe_concurrency_zero_test.toml");
|
||||
std::fs::write(&path, toml).unwrap();
|
||||
let err = ProxyConfig::load(&path).unwrap_err().to_string();
|
||||
assert!(err.contains("general.stun_nat_probe_concurrency must be > 0"));
|
||||
let _ = std::fs::remove_file(path);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn me_reinit_every_default_is_set() {
|
||||
let toml = r#"
|
||||
[censorship]
|
||||
tls_domain = "example.com"
|
||||
|
||||
[access.users]
|
||||
user = "00000000000000000000000000000000"
|
||||
"#;
|
||||
let dir = std::env::temp_dir();
|
||||
let path = dir.join("telemt_me_reinit_every_default_test.toml");
|
||||
std::fs::write(&path, toml).unwrap();
|
||||
let cfg = ProxyConfig::load(&path).unwrap();
|
||||
assert_eq!(
|
||||
cfg.general.me_reinit_every_secs,
|
||||
default_me_reinit_every_secs()
|
||||
);
|
||||
let _ = std::fs::remove_file(path);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn me_reinit_every_zero_is_rejected() {
|
||||
let toml = r#"
|
||||
[general]
|
||||
me_reinit_every_secs = 0
|
||||
|
||||
[censorship]
|
||||
tls_domain = "example.com"
|
||||
|
||||
[access.users]
|
||||
user = "00000000000000000000000000000000"
|
||||
"#;
|
||||
let dir = std::env::temp_dir();
|
||||
let path = dir.join("telemt_me_reinit_every_zero_test.toml");
|
||||
std::fs::write(&path, toml).unwrap();
|
||||
let err = ProxyConfig::load(&path).unwrap_err().to_string();
|
||||
assert!(err.contains("general.me_reinit_every_secs must be > 0"));
|
||||
let _ = std::fs::remove_file(path);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn upstream_connect_retry_attempts_zero_is_rejected() {
|
||||
let toml = r#"
|
||||
[general]
|
||||
upstream_connect_retry_attempts = 0
|
||||
|
||||
[censorship]
|
||||
tls_domain = "example.com"
|
||||
|
||||
[access.users]
|
||||
user = "00000000000000000000000000000000"
|
||||
"#;
|
||||
let dir = std::env::temp_dir();
|
||||
let path = dir.join("telemt_upstream_connect_retry_attempts_zero_test.toml");
|
||||
std::fs::write(&path, toml).unwrap();
|
||||
let err = ProxyConfig::load(&path).unwrap_err().to_string();
|
||||
assert!(err.contains("general.upstream_connect_retry_attempts must be > 0"));
|
||||
let _ = std::fs::remove_file(path);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn upstream_unhealthy_fail_threshold_zero_is_rejected() {
|
||||
let toml = r#"
|
||||
[general]
|
||||
upstream_unhealthy_fail_threshold = 0
|
||||
|
||||
[censorship]
|
||||
tls_domain = "example.com"
|
||||
|
||||
[access.users]
|
||||
user = "00000000000000000000000000000000"
|
||||
"#;
|
||||
let dir = std::env::temp_dir();
|
||||
let path = dir.join("telemt_upstream_unhealthy_fail_threshold_zero_test.toml");
|
||||
std::fs::write(&path, toml).unwrap();
|
||||
let err = ProxyConfig::load(&path).unwrap_err().to_string();
|
||||
assert!(err.contains("general.upstream_unhealthy_fail_threshold must be > 0"));
|
||||
let _ = std::fs::remove_file(path);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn me_hardswap_warmup_defaults_are_set() {
|
||||
let toml = r#"
|
||||
[censorship]
|
||||
tls_domain = "example.com"
|
||||
|
||||
[access.users]
|
||||
user = "00000000000000000000000000000000"
|
||||
"#;
|
||||
let dir = std::env::temp_dir();
|
||||
let path = dir.join("telemt_me_hardswap_warmup_defaults_test.toml");
|
||||
std::fs::write(&path, toml).unwrap();
|
||||
let cfg = ProxyConfig::load(&path).unwrap();
|
||||
assert_eq!(
|
||||
cfg.general.me_hardswap_warmup_delay_min_ms,
|
||||
default_me_hardswap_warmup_delay_min_ms()
|
||||
);
|
||||
assert_eq!(
|
||||
cfg.general.me_hardswap_warmup_delay_max_ms,
|
||||
default_me_hardswap_warmup_delay_max_ms()
|
||||
);
|
||||
assert_eq!(
|
||||
cfg.general.me_hardswap_warmup_extra_passes,
|
||||
default_me_hardswap_warmup_extra_passes()
|
||||
);
|
||||
assert_eq!(
|
||||
cfg.general.me_hardswap_warmup_pass_backoff_base_ms,
|
||||
default_me_hardswap_warmup_pass_backoff_base_ms()
|
||||
);
|
||||
let _ = std::fs::remove_file(path);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn me_hardswap_warmup_delay_range_is_validated() {
|
||||
let toml = r#"
|
||||
[general]
|
||||
me_hardswap_warmup_delay_min_ms = 2001
|
||||
me_hardswap_warmup_delay_max_ms = 2000
|
||||
|
||||
[censorship]
|
||||
tls_domain = "example.com"
|
||||
|
||||
[access.users]
|
||||
user = "00000000000000000000000000000000"
|
||||
"#;
|
||||
let dir = std::env::temp_dir();
|
||||
let path = dir.join("telemt_me_hardswap_warmup_delay_range_test.toml");
|
||||
std::fs::write(&path, toml).unwrap();
|
||||
let err = ProxyConfig::load(&path).unwrap_err().to_string();
|
||||
assert!(err.contains(
|
||||
"general.me_hardswap_warmup_delay_min_ms must be <= general.me_hardswap_warmup_delay_max_ms"
|
||||
));
|
||||
let _ = std::fs::remove_file(path);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn me_hardswap_warmup_delay_max_zero_is_rejected() {
|
||||
let toml = r#"
|
||||
[general]
|
||||
me_hardswap_warmup_delay_max_ms = 0
|
||||
|
||||
[censorship]
|
||||
tls_domain = "example.com"
|
||||
|
||||
[access.users]
|
||||
user = "00000000000000000000000000000000"
|
||||
"#;
|
||||
let dir = std::env::temp_dir();
|
||||
let path = dir.join("telemt_me_hardswap_warmup_delay_max_zero_test.toml");
|
||||
std::fs::write(&path, toml).unwrap();
|
||||
let err = ProxyConfig::load(&path).unwrap_err().to_string();
|
||||
assert!(err.contains("general.me_hardswap_warmup_delay_max_ms must be > 0"));
|
||||
let _ = std::fs::remove_file(path);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn me_hardswap_warmup_extra_passes_out_of_range_is_rejected() {
|
||||
let toml = r#"
|
||||
[general]
|
||||
me_hardswap_warmup_extra_passes = 11
|
||||
|
||||
[censorship]
|
||||
tls_domain = "example.com"
|
||||
|
||||
[access.users]
|
||||
user = "00000000000000000000000000000000"
|
||||
"#;
|
||||
let dir = std::env::temp_dir();
|
||||
let path = dir.join("telemt_me_hardswap_warmup_extra_passes_test.toml");
|
||||
std::fs::write(&path, toml).unwrap();
|
||||
let err = ProxyConfig::load(&path).unwrap_err().to_string();
|
||||
assert!(err.contains("general.me_hardswap_warmup_extra_passes must be within [0, 10]"));
|
||||
let _ = std::fs::remove_file(path);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn me_hardswap_warmup_pass_backoff_zero_is_rejected() {
|
||||
let toml = r#"
|
||||
[general]
|
||||
me_hardswap_warmup_pass_backoff_base_ms = 0
|
||||
|
||||
[censorship]
|
||||
tls_domain = "example.com"
|
||||
|
||||
[access.users]
|
||||
user = "00000000000000000000000000000000"
|
||||
"#;
|
||||
let dir = std::env::temp_dir();
|
||||
let path = dir.join("telemt_me_hardswap_warmup_backoff_zero_test.toml");
|
||||
std::fs::write(&path, toml).unwrap();
|
||||
let err = ProxyConfig::load(&path).unwrap_err().to_string();
|
||||
assert!(err.contains("general.me_hardswap_warmup_pass_backoff_base_ms must be > 0"));
|
||||
let _ = std::fs::remove_file(path);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn me_config_stable_snapshots_zero_is_rejected() {
|
||||
let toml = r#"
|
||||
[general]
|
||||
me_config_stable_snapshots = 0
|
||||
|
||||
[censorship]
|
||||
tls_domain = "example.com"
|
||||
|
||||
[access.users]
|
||||
user = "00000000000000000000000000000000"
|
||||
"#;
|
||||
let dir = std::env::temp_dir();
|
||||
let path = dir.join("telemt_me_config_stable_snapshots_zero_test.toml");
|
||||
std::fs::write(&path, toml).unwrap();
|
||||
let err = ProxyConfig::load(&path).unwrap_err().to_string();
|
||||
assert!(err.contains("general.me_config_stable_snapshots must be > 0"));
|
||||
let _ = std::fs::remove_file(path);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn proxy_secret_stable_snapshots_zero_is_rejected() {
|
||||
let toml = r#"
|
||||
[general]
|
||||
proxy_secret_stable_snapshots = 0
|
||||
|
||||
[censorship]
|
||||
tls_domain = "example.com"
|
||||
|
||||
[access.users]
|
||||
user = "00000000000000000000000000000000"
|
||||
"#;
|
||||
let dir = std::env::temp_dir();
|
||||
let path = dir.join("telemt_proxy_secret_stable_snapshots_zero_test.toml");
|
||||
std::fs::write(&path, toml).unwrap();
|
||||
let err = ProxyConfig::load(&path).unwrap_err().to_string();
|
||||
assert!(err.contains("general.proxy_secret_stable_snapshots must be > 0"));
|
||||
let _ = std::fs::remove_file(path);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn proxy_secret_len_max_out_of_range_is_rejected() {
|
||||
let toml = r#"
|
||||
[general]
|
||||
proxy_secret_len_max = 16
|
||||
|
||||
[censorship]
|
||||
tls_domain = "example.com"
|
||||
|
||||
[access.users]
|
||||
user = "00000000000000000000000000000000"
|
||||
"#;
|
||||
let dir = std::env::temp_dir();
|
||||
let path = dir.join("telemt_proxy_secret_len_max_out_of_range_test.toml");
|
||||
std::fs::write(&path, toml).unwrap();
|
||||
let err = ProxyConfig::load(&path).unwrap_err().to_string();
|
||||
assert!(err.contains("general.proxy_secret_len_max must be within [32, 4096]"));
|
||||
let _ = std::fs::remove_file(path);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn me_pool_min_fresh_ratio_out_of_range_is_rejected() {
|
||||
let toml = r#"
|
||||
@@ -498,4 +1056,87 @@ mod tests {
|
||||
assert_eq!(cfg.general.me_reinit_drain_timeout_secs, 90);
|
||||
let _ = std::fs::remove_file(path);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn invalid_ad_tag_is_disabled_during_load() {
|
||||
let toml = r#"
|
||||
[general]
|
||||
ad_tag = "not_hex"
|
||||
|
||||
[censorship]
|
||||
tls_domain = "example.com"
|
||||
|
||||
[access.users]
|
||||
user = "00000000000000000000000000000000"
|
||||
"#;
|
||||
let dir = std::env::temp_dir();
|
||||
let path = dir.join("telemt_invalid_ad_tag_test.toml");
|
||||
std::fs::write(&path, toml).unwrap();
|
||||
let cfg = ProxyConfig::load(&path).unwrap();
|
||||
assert!(cfg.general.ad_tag.is_none());
|
||||
let _ = std::fs::remove_file(path);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn valid_ad_tag_is_preserved_during_load() {
|
||||
let toml = r#"
|
||||
[general]
|
||||
ad_tag = "00112233445566778899aabbccddeeff"
|
||||
|
||||
[censorship]
|
||||
tls_domain = "example.com"
|
||||
|
||||
[access.users]
|
||||
user = "00000000000000000000000000000000"
|
||||
"#;
|
||||
let dir = std::env::temp_dir();
|
||||
let path = dir.join("telemt_valid_ad_tag_test.toml");
|
||||
std::fs::write(&path, toml).unwrap();
|
||||
let cfg = ProxyConfig::load(&path).unwrap();
|
||||
assert_eq!(
|
||||
cfg.general.ad_tag.as_deref(),
|
||||
Some("00112233445566778899aabbccddeeff")
|
||||
);
|
||||
let _ = std::fs::remove_file(path);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn invalid_dns_override_is_rejected() {
|
||||
let toml = r#"
|
||||
[network]
|
||||
dns_overrides = ["example.com:443:2001:db8::10"]
|
||||
|
||||
[censorship]
|
||||
tls_domain = "example.com"
|
||||
|
||||
[access.users]
|
||||
user = "00000000000000000000000000000000"
|
||||
"#;
|
||||
let dir = std::env::temp_dir();
|
||||
let path = dir.join("telemt_invalid_dns_override_test.toml");
|
||||
std::fs::write(&path, toml).unwrap();
|
||||
let err = ProxyConfig::load(&path).unwrap_err().to_string();
|
||||
assert!(err.contains("must be bracketed"));
|
||||
let _ = std::fs::remove_file(path);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn valid_dns_override_is_accepted() {
|
||||
let toml = r#"
|
||||
[network]
|
||||
dns_overrides = ["example.com:443:127.0.0.1", "example.net:443:[2001:db8::10]"]
|
||||
|
||||
[censorship]
|
||||
tls_domain = "example.com"
|
||||
|
||||
[access.users]
|
||||
user = "00000000000000000000000000000000"
|
||||
"#;
|
||||
let dir = std::env::temp_dir();
|
||||
let path = dir.join("telemt_valid_dns_override_test.toml");
|
||||
std::fs::write(&path, toml).unwrap();
|
||||
let cfg = ProxyConfig::load(&path).unwrap();
|
||||
assert_eq!(cfg.network.dns_overrides.len(), 2);
|
||||
let _ = std::fs::remove_file(path);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -59,6 +59,98 @@ impl std::fmt::Display for LogLevel {
|
||||
}
|
||||
}
|
||||
|
||||
/// Middle-End telemetry verbosity level.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum MeTelemetryLevel {
|
||||
#[default]
|
||||
Normal,
|
||||
Silent,
|
||||
Debug,
|
||||
}
|
||||
|
||||
impl MeTelemetryLevel {
|
||||
pub fn as_u8(self) -> u8 {
|
||||
match self {
|
||||
MeTelemetryLevel::Silent => 0,
|
||||
MeTelemetryLevel::Normal => 1,
|
||||
MeTelemetryLevel::Debug => 2,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn from_u8(raw: u8) -> Self {
|
||||
match raw {
|
||||
0 => MeTelemetryLevel::Silent,
|
||||
2 => MeTelemetryLevel::Debug,
|
||||
_ => MeTelemetryLevel::Normal,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn allows_normal(self) -> bool {
|
||||
!matches!(self, MeTelemetryLevel::Silent)
|
||||
}
|
||||
|
||||
pub fn allows_debug(self) -> bool {
|
||||
matches!(self, MeTelemetryLevel::Debug)
|
||||
}
|
||||
}
|
||||
|
||||
impl std::fmt::Display for MeTelemetryLevel {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
match self {
|
||||
MeTelemetryLevel::Silent => write!(f, "silent"),
|
||||
MeTelemetryLevel::Normal => write!(f, "normal"),
|
||||
MeTelemetryLevel::Debug => write!(f, "debug"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Middle-End SOCKS KDF fallback policy.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum MeSocksKdfPolicy {
|
||||
#[default]
|
||||
Strict,
|
||||
Compat,
|
||||
}
|
||||
|
||||
impl MeSocksKdfPolicy {
|
||||
pub fn as_u8(self) -> u8 {
|
||||
match self {
|
||||
MeSocksKdfPolicy::Strict => 0,
|
||||
MeSocksKdfPolicy::Compat => 1,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn from_u8(raw: u8) -> Self {
|
||||
match raw {
|
||||
1 => MeSocksKdfPolicy::Compat,
|
||||
_ => MeSocksKdfPolicy::Strict,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Telemetry controls for hot-path counters and ME diagnostics.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct TelemetryConfig {
|
||||
#[serde(default = "default_true")]
|
||||
pub core_enabled: bool,
|
||||
#[serde(default = "default_true")]
|
||||
pub user_enabled: bool,
|
||||
#[serde(default)]
|
||||
pub me_level: MeTelemetryLevel,
|
||||
}
|
||||
|
||||
impl Default for TelemetryConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
core_enabled: default_true(),
|
||||
user_enabled: default_true(),
|
||||
me_level: MeTelemetryLevel::Normal,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ============= Sub-Configs =============
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
@@ -76,7 +168,7 @@ impl Default for ProxyModes {
|
||||
Self {
|
||||
classic: false,
|
||||
secure: false,
|
||||
tls: true,
|
||||
tls: default_true(),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -87,7 +179,7 @@ pub struct NetworkConfig {
|
||||
pub ipv4: bool,
|
||||
|
||||
/// None = auto-detect IPv6 availability.
|
||||
#[serde(default)]
|
||||
#[serde(default = "default_network_ipv6")]
|
||||
pub ipv6: Option<bool>,
|
||||
|
||||
/// 4 or 6.
|
||||
@@ -97,12 +189,17 @@ pub struct NetworkConfig {
|
||||
#[serde(default)]
|
||||
pub multipath: bool,
|
||||
|
||||
/// Global switch for STUN probing.
|
||||
/// When false, STUN is fully disabled and only non-STUN detection remains.
|
||||
#[serde(default = "default_true")]
|
||||
pub stun_use: bool,
|
||||
|
||||
/// STUN servers list for public IP discovery.
|
||||
#[serde(default = "default_stun_servers")]
|
||||
pub stun_servers: Vec<String>,
|
||||
|
||||
/// Enable TCP STUN fallback when UDP is blocked.
|
||||
#[serde(default)]
|
||||
#[serde(default = "default_stun_tcp_fallback")]
|
||||
pub stun_tcp_fallback: bool,
|
||||
|
||||
/// HTTP-based public IP detection endpoints (fallback after STUN).
|
||||
@@ -112,19 +209,26 @@ pub struct NetworkConfig {
|
||||
/// Cache file path for detected public IP.
|
||||
#[serde(default = "default_cache_public_ip_path")]
|
||||
pub cache_public_ip_path: String,
|
||||
|
||||
/// Runtime DNS overrides in `host:port:ip` format.
|
||||
/// IPv6 IP values must be bracketed: `[2001:db8::1]`.
|
||||
#[serde(default)]
|
||||
pub dns_overrides: Vec<String>,
|
||||
}
|
||||
|
||||
impl Default for NetworkConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
ipv4: true,
|
||||
ipv6: Some(false),
|
||||
prefer: 4,
|
||||
ipv4: default_true(),
|
||||
ipv6: default_network_ipv6(),
|
||||
prefer: default_prefer_4(),
|
||||
multipath: false,
|
||||
stun_use: default_true(),
|
||||
stun_servers: default_stun_servers(),
|
||||
stun_tcp_fallback: true,
|
||||
stun_tcp_fallback: default_stun_tcp_fallback(),
|
||||
http_ip_detect_urls: default_http_ip_detect_urls(),
|
||||
cache_public_ip_path: default_cache_public_ip_path(),
|
||||
dns_overrides: Vec::new(),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -140,7 +244,7 @@ pub struct GeneralConfig {
|
||||
#[serde(default = "default_true")]
|
||||
pub fast_mode: bool,
|
||||
|
||||
#[serde(default)]
|
||||
#[serde(default = "default_true")]
|
||||
pub use_middle_proxy: bool,
|
||||
|
||||
#[serde(default)]
|
||||
@@ -148,7 +252,7 @@ pub struct GeneralConfig {
|
||||
|
||||
/// Path to proxy-secret binary file (auto-downloaded if absent).
|
||||
/// Infrastructure secret from https://core.telegram.org/getProxySecret.
|
||||
#[serde(default)]
|
||||
#[serde(default = "default_proxy_secret_path")]
|
||||
pub proxy_secret_path: Option<String>,
|
||||
|
||||
/// Public IP override for middle-proxy NAT environments.
|
||||
@@ -157,23 +261,29 @@ pub struct GeneralConfig {
|
||||
pub middle_proxy_nat_ip: Option<IpAddr>,
|
||||
|
||||
/// Enable STUN-based NAT probing to discover public IP:port for ME KDF.
|
||||
#[serde(default)]
|
||||
#[serde(default = "default_true")]
|
||||
pub middle_proxy_nat_probe: bool,
|
||||
|
||||
/// Optional STUN server address (host:port) for NAT probing.
|
||||
#[serde(default)]
|
||||
/// Deprecated legacy single STUN server for NAT probing.
|
||||
/// Use `network.stun_servers` instead.
|
||||
#[serde(default = "default_middle_proxy_nat_stun")]
|
||||
pub middle_proxy_nat_stun: Option<String>,
|
||||
|
||||
/// Optional list of STUN servers for NAT probing fallback.
|
||||
#[serde(default)]
|
||||
/// Deprecated legacy STUN list for NAT probing fallback.
|
||||
/// Use `network.stun_servers` instead.
|
||||
#[serde(default = "default_middle_proxy_nat_stun_servers")]
|
||||
pub middle_proxy_nat_stun_servers: Vec<String>,
|
||||
|
||||
/// Maximum number of concurrent STUN probes during NAT detection.
|
||||
#[serde(default = "default_stun_nat_probe_concurrency")]
|
||||
pub stun_nat_probe_concurrency: usize,
|
||||
|
||||
/// Desired size of active Middle-Proxy writer pool.
|
||||
#[serde(default = "default_pool_size")]
|
||||
pub middle_proxy_pool_size: usize,
|
||||
|
||||
/// Number of warm standby ME connections kept pre-initialized.
|
||||
#[serde(default)]
|
||||
#[serde(default = "default_middle_proxy_warm_standby")]
|
||||
pub middle_proxy_warm_standby: usize,
|
||||
|
||||
/// Enable ME keepalive padding frames.
|
||||
@@ -206,6 +316,22 @@ pub struct GeneralConfig {
|
||||
#[serde(default = "default_desync_all_full")]
|
||||
pub desync_all_full: bool,
|
||||
|
||||
/// Enable per-IP forensic observation buckets for scanners and handshake failures.
|
||||
#[serde(default = "default_true")]
|
||||
pub beobachten: bool,
|
||||
|
||||
/// Observation retention window in minutes for per-IP forensic buckets.
|
||||
#[serde(default = "default_beobachten_minutes")]
|
||||
pub beobachten_minutes: u64,
|
||||
|
||||
/// Snapshot flush interval in seconds for beob output file.
|
||||
#[serde(default = "default_beobachten_flush_secs")]
|
||||
pub beobachten_flush_secs: u64,
|
||||
|
||||
/// Snapshot file path for beob output.
|
||||
#[serde(default = "default_beobachten_file")]
|
||||
pub beobachten_file: String,
|
||||
|
||||
/// Enable C-like hard-swap for ME pool generations.
|
||||
/// When true, Telemt prewarms a new generation and switches once full coverage is reached.
|
||||
#[serde(default = "default_hardswap")]
|
||||
@@ -224,7 +350,7 @@ pub struct GeneralConfig {
|
||||
pub me_warmup_step_jitter_ms: u64,
|
||||
|
||||
/// Max concurrent reconnect attempts per DC.
|
||||
#[serde(default)]
|
||||
#[serde(default = "default_me_reconnect_max_concurrent_per_dc")]
|
||||
pub me_reconnect_max_concurrent_per_dc: u32,
|
||||
|
||||
/// Base backoff in ms for reconnect.
|
||||
@@ -236,9 +362,21 @@ pub struct GeneralConfig {
|
||||
pub me_reconnect_backoff_cap_ms: u64,
|
||||
|
||||
/// Fast retry attempts before backoff.
|
||||
#[serde(default)]
|
||||
#[serde(default = "default_me_reconnect_fast_retry_count")]
|
||||
pub me_reconnect_fast_retry_count: u32,
|
||||
|
||||
/// Connect attempts for the selected upstream before returning error/fallback.
|
||||
#[serde(default = "default_upstream_connect_retry_attempts")]
|
||||
pub upstream_connect_retry_attempts: u32,
|
||||
|
||||
/// Delay in milliseconds between upstream connect attempts.
|
||||
#[serde(default = "default_upstream_connect_retry_backoff_ms")]
|
||||
pub upstream_connect_retry_backoff_ms: u64,
|
||||
|
||||
/// Consecutive failed requests before upstream is marked unhealthy.
|
||||
#[serde(default = "default_upstream_unhealthy_fail_threshold")]
|
||||
pub upstream_unhealthy_fail_threshold: u32,
|
||||
|
||||
/// Ignore STUN/interface IP mismatch (keep using Middle Proxy even if NAT detected).
|
||||
#[serde(default)]
|
||||
pub stun_iface_mismatch_ignore: bool,
|
||||
@@ -254,6 +392,26 @@ pub struct GeneralConfig {
|
||||
#[serde(default)]
|
||||
pub disable_colors: bool,
|
||||
|
||||
/// Runtime telemetry controls for counters/metrics in hot paths.
|
||||
#[serde(default)]
|
||||
pub telemetry: TelemetryConfig,
|
||||
|
||||
/// SOCKS-bound KDF policy for Middle-End handshake.
|
||||
#[serde(default)]
|
||||
pub me_socks_kdf_policy: MeSocksKdfPolicy,
|
||||
|
||||
/// Base backpressure timeout in milliseconds for ME route channel send.
|
||||
#[serde(default = "default_me_route_backpressure_base_timeout_ms")]
|
||||
pub me_route_backpressure_base_timeout_ms: u64,
|
||||
|
||||
/// High backpressure timeout in milliseconds when queue occupancy is above watermark.
|
||||
#[serde(default = "default_me_route_backpressure_high_timeout_ms")]
|
||||
pub me_route_backpressure_high_timeout_ms: u64,
|
||||
|
||||
/// Queue occupancy percent threshold for high backpressure timeout.
|
||||
#[serde(default = "default_me_route_backpressure_high_watermark_pct")]
|
||||
pub me_route_backpressure_high_watermark_pct: u8,
|
||||
|
||||
/// [general.links] — proxy link generation overrides.
|
||||
#[serde(default)]
|
||||
pub links: LinksConfig,
|
||||
@@ -264,9 +422,49 @@ pub struct GeneralConfig {
|
||||
|
||||
/// Unified ME updater interval in seconds for getProxyConfig/getProxyConfigV6/getProxySecret.
|
||||
/// When omitted, effective value falls back to legacy proxy_*_auto_reload_secs fields.
|
||||
#[serde(default)]
|
||||
#[serde(default = "default_update_every")]
|
||||
pub update_every: Option<u64>,
|
||||
|
||||
/// Periodic ME pool reinitialization interval in seconds.
|
||||
#[serde(default = "default_me_reinit_every_secs")]
|
||||
pub me_reinit_every_secs: u64,
|
||||
|
||||
/// Minimum delay in ms between hardswap warmup connect attempts.
|
||||
#[serde(default = "default_me_hardswap_warmup_delay_min_ms")]
|
||||
pub me_hardswap_warmup_delay_min_ms: u64,
|
||||
|
||||
/// Maximum delay in ms between hardswap warmup connect attempts.
|
||||
#[serde(default = "default_me_hardswap_warmup_delay_max_ms")]
|
||||
pub me_hardswap_warmup_delay_max_ms: u64,
|
||||
|
||||
/// Additional warmup passes in the same hardswap cycle after the base pass.
|
||||
#[serde(default = "default_me_hardswap_warmup_extra_passes")]
|
||||
pub me_hardswap_warmup_extra_passes: u8,
|
||||
|
||||
/// Base backoff in ms between hardswap warmup passes when floor is still incomplete.
|
||||
#[serde(default = "default_me_hardswap_warmup_pass_backoff_base_ms")]
|
||||
pub me_hardswap_warmup_pass_backoff_base_ms: u64,
|
||||
|
||||
/// Number of identical getProxyConfig snapshots required before applying ME map updates.
|
||||
#[serde(default = "default_me_config_stable_snapshots")]
|
||||
pub me_config_stable_snapshots: u8,
|
||||
|
||||
/// Cooldown in seconds between applied ME map updates.
|
||||
#[serde(default = "default_me_config_apply_cooldown_secs")]
|
||||
pub me_config_apply_cooldown_secs: u64,
|
||||
|
||||
/// Number of identical getProxySecret snapshots required before runtime secret rotation.
|
||||
#[serde(default = "default_proxy_secret_stable_snapshots")]
|
||||
pub proxy_secret_stable_snapshots: u8,
|
||||
|
||||
/// Enable runtime proxy-secret rotation from getProxySecret.
|
||||
#[serde(default = "default_proxy_secret_rotate_runtime")]
|
||||
pub proxy_secret_rotate_runtime: bool,
|
||||
|
||||
/// Maximum allowed proxy-secret length in bytes for startup and runtime refresh.
|
||||
#[serde(default = "default_proxy_secret_len_max")]
|
||||
pub proxy_secret_len_max: usize,
|
||||
|
||||
/// Drain-TTL in seconds for stale ME writers after endpoint map changes.
|
||||
/// During TTL, stale writers may be used only as fallback for new bindings.
|
||||
#[serde(default = "default_me_pool_drain_ttl_secs")]
|
||||
@@ -314,38 +512,61 @@ impl Default for GeneralConfig {
|
||||
Self {
|
||||
modes: ProxyModes::default(),
|
||||
prefer_ipv6: false,
|
||||
fast_mode: true,
|
||||
use_middle_proxy: false,
|
||||
fast_mode: default_true(),
|
||||
use_middle_proxy: default_true(),
|
||||
ad_tag: None,
|
||||
proxy_secret_path: None,
|
||||
proxy_secret_path: default_proxy_secret_path(),
|
||||
middle_proxy_nat_ip: None,
|
||||
middle_proxy_nat_probe: false,
|
||||
middle_proxy_nat_stun: None,
|
||||
middle_proxy_nat_stun_servers: Vec::new(),
|
||||
middle_proxy_nat_probe: default_true(),
|
||||
middle_proxy_nat_stun: default_middle_proxy_nat_stun(),
|
||||
middle_proxy_nat_stun_servers: default_middle_proxy_nat_stun_servers(),
|
||||
stun_nat_probe_concurrency: default_stun_nat_probe_concurrency(),
|
||||
middle_proxy_pool_size: default_pool_size(),
|
||||
middle_proxy_warm_standby: 16,
|
||||
me_keepalive_enabled: true,
|
||||
middle_proxy_warm_standby: default_middle_proxy_warm_standby(),
|
||||
me_keepalive_enabled: default_true(),
|
||||
me_keepalive_interval_secs: default_keepalive_interval(),
|
||||
me_keepalive_jitter_secs: default_keepalive_jitter(),
|
||||
me_keepalive_payload_random: true,
|
||||
me_warmup_stagger_enabled: true,
|
||||
me_keepalive_payload_random: default_true(),
|
||||
me_warmup_stagger_enabled: default_true(),
|
||||
me_warmup_step_delay_ms: default_warmup_step_delay_ms(),
|
||||
me_warmup_step_jitter_ms: default_warmup_step_jitter_ms(),
|
||||
me_reconnect_max_concurrent_per_dc: 8,
|
||||
me_reconnect_max_concurrent_per_dc: default_me_reconnect_max_concurrent_per_dc(),
|
||||
me_reconnect_backoff_base_ms: default_reconnect_backoff_base_ms(),
|
||||
me_reconnect_backoff_cap_ms: default_reconnect_backoff_cap_ms(),
|
||||
me_reconnect_fast_retry_count: 8,
|
||||
me_reconnect_fast_retry_count: default_me_reconnect_fast_retry_count(),
|
||||
upstream_connect_retry_attempts: default_upstream_connect_retry_attempts(),
|
||||
upstream_connect_retry_backoff_ms: default_upstream_connect_retry_backoff_ms(),
|
||||
upstream_unhealthy_fail_threshold: default_upstream_unhealthy_fail_threshold(),
|
||||
stun_iface_mismatch_ignore: false,
|
||||
unknown_dc_log_path: default_unknown_dc_log_path(),
|
||||
log_level: LogLevel::Normal,
|
||||
disable_colors: false,
|
||||
telemetry: TelemetryConfig::default(),
|
||||
me_socks_kdf_policy: MeSocksKdfPolicy::Strict,
|
||||
me_route_backpressure_base_timeout_ms: default_me_route_backpressure_base_timeout_ms(),
|
||||
me_route_backpressure_high_timeout_ms: default_me_route_backpressure_high_timeout_ms(),
|
||||
me_route_backpressure_high_watermark_pct: default_me_route_backpressure_high_watermark_pct(),
|
||||
links: LinksConfig::default(),
|
||||
crypto_pending_buffer: default_crypto_pending_buffer(),
|
||||
max_client_frame: default_max_client_frame(),
|
||||
desync_all_full: default_desync_all_full(),
|
||||
beobachten: default_true(),
|
||||
beobachten_minutes: default_beobachten_minutes(),
|
||||
beobachten_flush_secs: default_beobachten_flush_secs(),
|
||||
beobachten_file: default_beobachten_file(),
|
||||
hardswap: default_hardswap(),
|
||||
fast_mode_min_tls_record: default_fast_mode_min_tls_record(),
|
||||
update_every: Some(default_update_every_secs()),
|
||||
update_every: default_update_every(),
|
||||
me_reinit_every_secs: default_me_reinit_every_secs(),
|
||||
me_hardswap_warmup_delay_min_ms: default_me_hardswap_warmup_delay_min_ms(),
|
||||
me_hardswap_warmup_delay_max_ms: default_me_hardswap_warmup_delay_max_ms(),
|
||||
me_hardswap_warmup_extra_passes: default_me_hardswap_warmup_extra_passes(),
|
||||
me_hardswap_warmup_pass_backoff_base_ms: default_me_hardswap_warmup_pass_backoff_base_ms(),
|
||||
me_config_stable_snapshots: default_me_config_stable_snapshots(),
|
||||
me_config_apply_cooldown_secs: default_me_config_apply_cooldown_secs(),
|
||||
proxy_secret_stable_snapshots: default_proxy_secret_stable_snapshots(),
|
||||
proxy_secret_rotate_runtime: default_proxy_secret_rotate_runtime(),
|
||||
proxy_secret_len_max: default_proxy_secret_len_max(),
|
||||
me_pool_drain_ttl_secs: default_me_pool_drain_ttl_secs(),
|
||||
me_pool_min_fresh_ratio: default_me_pool_min_fresh_ratio(),
|
||||
me_reinit_drain_timeout_secs: default_me_reinit_drain_timeout_secs(),
|
||||
@@ -353,7 +574,7 @@ impl Default for GeneralConfig {
|
||||
proxy_config_auto_reload_secs: default_proxy_config_reload_secs(),
|
||||
ntp_check: default_ntp_check(),
|
||||
ntp_servers: default_ntp_servers(),
|
||||
auto_degradation_enabled: true,
|
||||
auto_degradation_enabled: default_true(),
|
||||
degradation_min_unavailable_dc_groups: default_degradation_min_unavailable_dc_groups(),
|
||||
}
|
||||
}
|
||||
@@ -367,6 +588,11 @@ impl GeneralConfig {
|
||||
.unwrap_or_else(|| self.proxy_secret_auto_reload_secs.min(self.proxy_config_auto_reload_secs))
|
||||
}
|
||||
|
||||
/// Resolve periodic zero-downtime reinit interval for ME writers.
|
||||
pub fn effective_me_reinit_every_secs(&self) -> u64 {
|
||||
self.me_reinit_every_secs
|
||||
}
|
||||
|
||||
/// Resolve force-close timeout for stale writers.
|
||||
/// `me_reinit_drain_timeout_secs` remains backward-compatible alias.
|
||||
pub fn effective_me_pool_force_close_secs(&self) -> u64 {
|
||||
@@ -375,11 +601,11 @@ impl GeneralConfig {
|
||||
}
|
||||
|
||||
/// `[general.links]` — proxy link generation settings.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct LinksConfig {
|
||||
/// List of usernames whose tg:// links to display at startup.
|
||||
/// `"*"` = all users, `["alice", "bob"]` = specific users.
|
||||
#[serde(default)]
|
||||
#[serde(default = "default_links_show")]
|
||||
pub show: ShowLink,
|
||||
|
||||
/// Public hostname/IP for tg:// link generation (overrides detected IP).
|
||||
@@ -391,15 +617,25 @@ pub struct LinksConfig {
|
||||
pub public_port: Option<u16>,
|
||||
}
|
||||
|
||||
impl Default for LinksConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
show: default_links_show(),
|
||||
public_host: None,
|
||||
public_port: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct ServerConfig {
|
||||
#[serde(default = "default_port")]
|
||||
pub port: u16,
|
||||
|
||||
#[serde(default)]
|
||||
#[serde(default = "default_listen_addr_ipv4")]
|
||||
pub listen_addr_ipv4: Option<String>,
|
||||
|
||||
#[serde(default)]
|
||||
#[serde(default = "default_listen_addr_ipv6_opt")]
|
||||
pub listen_addr_ipv6: Option<String>,
|
||||
|
||||
#[serde(default)]
|
||||
@@ -434,8 +670,8 @@ impl Default for ServerConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
port: default_port(),
|
||||
listen_addr_ipv4: Some(default_listen_addr()),
|
||||
listen_addr_ipv6: Some("::".to_string()),
|
||||
listen_addr_ipv4: default_listen_addr_ipv4(),
|
||||
listen_addr_ipv6: default_listen_addr_ipv6_opt(),
|
||||
listen_unix_sock: None,
|
||||
listen_unix_sock_perm: None,
|
||||
listen_tcp: None,
|
||||
@@ -508,7 +744,7 @@ pub struct AntiCensorshipConfig {
|
||||
pub fake_cert_len: usize,
|
||||
|
||||
/// Enable TLS certificate emulation using cached real certificates.
|
||||
#[serde(default)]
|
||||
#[serde(default = "default_true")]
|
||||
pub tls_emulation: bool,
|
||||
|
||||
/// Directory to store TLS front cache (on disk).
|
||||
@@ -536,6 +772,12 @@ pub struct AntiCensorshipConfig {
|
||||
/// Enforce ALPN echo of client preference.
|
||||
#[serde(default = "default_alpn_enforce")]
|
||||
pub alpn_enforce: bool,
|
||||
|
||||
/// Send PROXY protocol header when connecting to mask_host.
|
||||
/// 0 = disabled, 1 = v1 (text), 2 = v2 (binary).
|
||||
/// Allows the backend to see the real client IP.
|
||||
#[serde(default)]
|
||||
pub mask_proxy_protocol: u8,
|
||||
}
|
||||
|
||||
impl Default for AntiCensorshipConfig {
|
||||
@@ -543,25 +785,26 @@ impl Default for AntiCensorshipConfig {
|
||||
Self {
|
||||
tls_domain: default_tls_domain(),
|
||||
tls_domains: Vec::new(),
|
||||
mask: true,
|
||||
mask: default_true(),
|
||||
mask_host: None,
|
||||
mask_port: default_mask_port(),
|
||||
mask_unix_sock: None,
|
||||
fake_cert_len: default_fake_cert_len(),
|
||||
tls_emulation: false,
|
||||
tls_emulation: true,
|
||||
tls_front_dir: default_tls_front_dir(),
|
||||
server_hello_delay_min_ms: default_server_hello_delay_min_ms(),
|
||||
server_hello_delay_max_ms: default_server_hello_delay_max_ms(),
|
||||
tls_new_session_tickets: default_tls_new_session_tickets(),
|
||||
tls_full_cert_ttl_secs: default_tls_full_cert_ttl_secs(),
|
||||
alpn_enforce: default_alpn_enforce(),
|
||||
mask_proxy_protocol: 0,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
pub struct AccessConfig {
|
||||
#[serde(default)]
|
||||
#[serde(default = "default_access_users")]
|
||||
pub users: HashMap<String, String>,
|
||||
|
||||
#[serde(default)]
|
||||
@@ -588,13 +831,8 @@ pub struct AccessConfig {
|
||||
|
||||
impl Default for AccessConfig {
|
||||
fn default() -> Self {
|
||||
let mut users = HashMap::new();
|
||||
users.insert(
|
||||
"default".to_string(),
|
||||
"00000000000000000000000000000000".to_string(),
|
||||
);
|
||||
Self {
|
||||
users,
|
||||
users: default_access_users(),
|
||||
user_max_tcp_conns: HashMap::new(),
|
||||
user_expirations: HashMap::new(),
|
||||
user_data_quota: HashMap::new(),
|
||||
@@ -676,10 +914,11 @@ pub struct ListenerConfig {
|
||||
/// In TOML, this can be:
|
||||
/// - `show_link = "*"` — show links for all users
|
||||
/// - `show_link = ["a", "b"]` — show links for specific users
|
||||
/// - omitted — show no links (default)
|
||||
#[derive(Debug, Clone)]
|
||||
/// - omitted — default depends on the owning config field
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub enum ShowLink {
|
||||
/// Don't show any links (default when omitted).
|
||||
#[default]
|
||||
None,
|
||||
/// Show links for all configured users.
|
||||
All,
|
||||
@@ -687,10 +926,8 @@ pub enum ShowLink {
|
||||
Specific(Vec<String>),
|
||||
}
|
||||
|
||||
impl Default for ShowLink {
|
||||
fn default() -> Self {
|
||||
ShowLink::None
|
||||
}
|
||||
fn default_links_show() -> ShowLink {
|
||||
ShowLink::All
|
||||
}
|
||||
|
||||
impl ShowLink {
|
||||
|
||||
@@ -11,6 +11,8 @@
|
||||
//! `HandshakeSuccess`, `ObfuscationParams`) are responsible for
|
||||
//! zeroizing their own copies.
|
||||
|
||||
#![allow(dead_code)]
|
||||
|
||||
use aes::Aes256;
|
||||
use ctr::{Ctr128BE, cipher::{KeyIvInit, StreamCipher}};
|
||||
use zeroize::Zeroize;
|
||||
@@ -21,13 +23,13 @@ type Aes256Ctr = Ctr128BE<Aes256>;
|
||||
// ============= AES-256-CTR =============
|
||||
|
||||
/// AES-256-CTR encryptor/decryptor
|
||||
///
|
||||
///
|
||||
/// CTR mode is symmetric — encryption and decryption are the same operation.
|
||||
///
|
||||
/// **Zeroize note:** The inner `Aes256Ctr` cipher state (expanded key schedule
|
||||
/// + counter) is opaque and cannot be zeroized. If you need to protect key
|
||||
/// material, zeroize the `[u8; 32]` key and `u128` IV at the call site
|
||||
/// before dropping them.
|
||||
/// + counter) is opaque and cannot be zeroized. If you need to protect key
|
||||
/// material, zeroize the `[u8; 32]` key and `u128` IV at the call site
|
||||
/// before dropping them.
|
||||
pub struct AesCtr {
|
||||
cipher: Aes256Ctr,
|
||||
}
|
||||
@@ -147,7 +149,7 @@ impl AesCbc {
|
||||
///
|
||||
/// CBC Encryption: C[i] = AES_Encrypt(P[i] XOR C[i-1]), where C[-1] = IV
|
||||
pub fn encrypt(&self, data: &[u8]) -> Result<Vec<u8>> {
|
||||
if data.len() % Self::BLOCK_SIZE != 0 {
|
||||
if !data.len().is_multiple_of(Self::BLOCK_SIZE) {
|
||||
return Err(ProxyError::Crypto(
|
||||
format!("CBC data must be aligned to 16 bytes, got {}", data.len())
|
||||
));
|
||||
@@ -178,7 +180,7 @@ impl AesCbc {
|
||||
///
|
||||
/// CBC Decryption: P[i] = AES_Decrypt(C[i]) XOR C[i-1], where C[-1] = IV
|
||||
pub fn decrypt(&self, data: &[u8]) -> Result<Vec<u8>> {
|
||||
if data.len() % Self::BLOCK_SIZE != 0 {
|
||||
if !data.len().is_multiple_of(Self::BLOCK_SIZE) {
|
||||
return Err(ProxyError::Crypto(
|
||||
format!("CBC data must be aligned to 16 bytes, got {}", data.len())
|
||||
));
|
||||
@@ -207,7 +209,7 @@ impl AesCbc {
|
||||
|
||||
/// Encrypt data in-place
|
||||
pub fn encrypt_in_place(&self, data: &mut [u8]) -> Result<()> {
|
||||
if data.len() % Self::BLOCK_SIZE != 0 {
|
||||
if !data.len().is_multiple_of(Self::BLOCK_SIZE) {
|
||||
return Err(ProxyError::Crypto(
|
||||
format!("CBC data must be aligned to 16 bytes, got {}", data.len())
|
||||
));
|
||||
@@ -240,7 +242,7 @@ impl AesCbc {
|
||||
|
||||
/// Decrypt data in-place
|
||||
pub fn decrypt_in_place(&self, data: &mut [u8]) -> Result<()> {
|
||||
if data.len() % Self::BLOCK_SIZE != 0 {
|
||||
if !data.len().is_multiple_of(Self::BLOCK_SIZE) {
|
||||
return Err(ProxyError::Crypto(
|
||||
format!("CBC data must be aligned to 16 bytes, got {}", data.len())
|
||||
));
|
||||
|
||||
@@ -64,6 +64,7 @@ pub fn crc32c(data: &[u8]) -> u32 {
|
||||
///
|
||||
/// Returned buffer layout (IPv4):
|
||||
/// nonce_srv | nonce_clt | clt_ts | srv_ip | clt_port | purpose | clt_ip | srv_port | secret | nonce_srv | [clt_v6 | srv_v6] | nonce_clt
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub fn build_middleproxy_prekey(
|
||||
nonce_srv: &[u8; 16],
|
||||
nonce_clt: &[u8; 16],
|
||||
@@ -108,6 +109,7 @@ pub fn build_middleproxy_prekey(
|
||||
/// Uses MD5 + SHA-1 as mandated by the Telegram Middle Proxy protocol.
|
||||
/// These algorithms are NOT replaceable here — changing them would break
|
||||
/// interoperability with Telegram's middle proxy infrastructure.
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub fn derive_middleproxy_keys(
|
||||
nonce_srv: &[u8; 16],
|
||||
nonce_clt: &[u8; 16],
|
||||
|
||||
@@ -6,7 +6,6 @@ pub mod random;
|
||||
|
||||
pub use aes::{AesCtr, AesCbc};
|
||||
pub use hash::{
|
||||
build_middleproxy_prekey, crc32, crc32c, derive_middleproxy_keys, md5, sha1, sha256,
|
||||
sha256_hmac,
|
||||
build_middleproxy_prekey, crc32, crc32c, derive_middleproxy_keys, sha256, sha256_hmac,
|
||||
};
|
||||
pub use random::SecureRandom;
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
//! Pseudorandom
|
||||
|
||||
#![allow(deprecated)]
|
||||
#![allow(dead_code)]
|
||||
|
||||
use rand::{Rng, RngCore, SeedableRng};
|
||||
use rand::rngs::StdRng;
|
||||
use parking_lot::Mutex;
|
||||
@@ -92,7 +95,7 @@ impl SecureRandom {
|
||||
return 0;
|
||||
}
|
||||
|
||||
let bytes_needed = (k + 7) / 8;
|
||||
let bytes_needed = k.div_ceil(8);
|
||||
let bytes = self.bytes(bytes_needed.min(8));
|
||||
|
||||
let mut result = 0u64;
|
||||
|
||||
13
src/error.rs
13
src/error.rs
@@ -1,5 +1,7 @@
|
||||
//! Error Types
|
||||
|
||||
#![allow(dead_code)]
|
||||
|
||||
use std::fmt;
|
||||
use std::net::SocketAddr;
|
||||
use thiserror::Error;
|
||||
@@ -89,7 +91,7 @@ impl From<StreamError> for std::io::Error {
|
||||
std::io::Error::new(std::io::ErrorKind::UnexpectedEof, err)
|
||||
}
|
||||
StreamError::Poisoned { .. } => {
|
||||
std::io::Error::new(std::io::ErrorKind::Other, err)
|
||||
std::io::Error::other(err)
|
||||
}
|
||||
StreamError::BufferOverflow { .. } => {
|
||||
std::io::Error::new(std::io::ErrorKind::OutOfMemory, err)
|
||||
@@ -98,7 +100,7 @@ impl From<StreamError> for std::io::Error {
|
||||
std::io::Error::new(std::io::ErrorKind::InvalidData, err)
|
||||
}
|
||||
StreamError::PartialRead { .. } | StreamError::PartialWrite { .. } => {
|
||||
std::io::Error::new(std::io::ErrorKind::Other, err)
|
||||
std::io::Error::other(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -133,12 +135,7 @@ impl Recoverable for StreamError {
|
||||
}
|
||||
|
||||
fn can_continue(&self) -> bool {
|
||||
match self {
|
||||
Self::Poisoned { .. } => false,
|
||||
Self::UnexpectedEof => false,
|
||||
Self::BufferOverflow { .. } => false,
|
||||
_ => true,
|
||||
}
|
||||
!matches!(self, Self::Poisoned { .. } | Self::UnexpectedEof | Self::BufferOverflow { .. })
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
// src/ip_tracker.rs
|
||||
// Модуль для отслеживания и ограничения уникальных IP-адресов пользователей
|
||||
// IP address tracking and limiting for users
|
||||
|
||||
#![allow(dead_code)]
|
||||
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::net::IpAddr;
|
||||
|
||||
478
src/main.rs
478
src/main.rs
@@ -1,5 +1,7 @@
|
||||
//! telemt — Telegram MTProto Proxy
|
||||
|
||||
#![allow(unused_assignments)]
|
||||
|
||||
use std::net::SocketAddr;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
@@ -33,10 +35,13 @@ use crate::crypto::SecureRandom;
|
||||
use crate::ip_tracker::UserIpTracker;
|
||||
use crate::network::probe::{decide_network_capabilities, log_probe_result, run_probe};
|
||||
use crate::proxy::ClientHandler;
|
||||
use crate::stats::beobachten::BeobachtenStore;
|
||||
use crate::stats::telemetry::TelemetryPolicy;
|
||||
use crate::stats::{ReplayChecker, Stats};
|
||||
use crate::stream::BufferPool;
|
||||
use crate::transport::middle_proxy::{
|
||||
MePool, fetch_proxy_config, run_me_ping, MePingFamily, MePingSample, format_sample_line,
|
||||
format_me_route,
|
||||
};
|
||||
use crate::transport::{ListenOptions, UpstreamManager, create_listener, find_listener_processes};
|
||||
use crate::tls_front::TlsFrontCache;
|
||||
@@ -157,6 +162,15 @@ fn print_proxy_links(host: &str, port: u16, config: &ProxyConfig) {
|
||||
info!(target: "telemt::links", "------------------------");
|
||||
}
|
||||
|
||||
async fn write_beobachten_snapshot(path: &str, payload: &str) -> std::io::Result<()> {
|
||||
if let Some(parent) = std::path::Path::new(path).parent()
|
||||
&& !parent.as_os_str().is_empty()
|
||||
{
|
||||
tokio::fs::create_dir_all(parent).await?;
|
||||
}
|
||||
tokio::fs::write(path, payload).await
|
||||
}
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() -> std::result::Result<(), Box<dyn std::error::Error>> {
|
||||
let (config_path, cli_silent, cli_log_level) = parse_cli();
|
||||
@@ -181,6 +195,11 @@ async fn main() -> std::result::Result<(), Box<dyn std::error::Error>> {
|
||||
std::process::exit(1);
|
||||
}
|
||||
|
||||
if let Err(e) = crate::network::dns_overrides::install_entries(&config.network.dns_overrides) {
|
||||
eprintln!("[telemt] Invalid network.dns_overrides: {}", e);
|
||||
std::process::exit(1);
|
||||
}
|
||||
|
||||
let has_rust_log = std::env::var("RUST_LOG").is_ok();
|
||||
let effective_log_level = if cli_silent {
|
||||
LogLevel::Silent
|
||||
@@ -191,14 +210,14 @@ async fn main() -> std::result::Result<(), Box<dyn std::error::Error>> {
|
||||
};
|
||||
|
||||
let (filter_layer, filter_handle) = reload::Layer::new(EnvFilter::new("info"));
|
||||
|
||||
|
||||
// Configure color output based on config
|
||||
let fmt_layer = if config.general.disable_colors {
|
||||
fmt::Layer::default().with_ansi(false)
|
||||
} else {
|
||||
fmt::Layer::default().with_ansi(true)
|
||||
};
|
||||
|
||||
|
||||
tracing_subscriber::registry()
|
||||
.with(filter_layer)
|
||||
.with(fmt_layer)
|
||||
@@ -242,10 +261,149 @@ async fn main() -> std::result::Result<(), Box<dyn std::error::Error>> {
|
||||
warn!("Using default tls_domain. Consider setting a custom domain.");
|
||||
}
|
||||
|
||||
let upstream_manager = Arc::new(UpstreamManager::new(
|
||||
config.upstreams.clone(),
|
||||
config.general.upstream_connect_retry_attempts,
|
||||
config.general.upstream_connect_retry_backoff_ms,
|
||||
config.general.upstream_unhealthy_fail_threshold,
|
||||
));
|
||||
|
||||
let mut tls_domains = Vec::with_capacity(1 + config.censorship.tls_domains.len());
|
||||
tls_domains.push(config.censorship.tls_domain.clone());
|
||||
for d in &config.censorship.tls_domains {
|
||||
if !tls_domains.contains(d) {
|
||||
tls_domains.push(d.clone());
|
||||
}
|
||||
}
|
||||
|
||||
// Start TLS front fetching in background immediately, in parallel with STUN probing.
|
||||
let tls_cache: Option<Arc<TlsFrontCache>> = if config.censorship.tls_emulation {
|
||||
let cache = Arc::new(TlsFrontCache::new(
|
||||
&tls_domains,
|
||||
config.censorship.fake_cert_len,
|
||||
&config.censorship.tls_front_dir,
|
||||
));
|
||||
cache.load_from_disk().await;
|
||||
|
||||
let port = config.censorship.mask_port;
|
||||
let proxy_protocol = config.censorship.mask_proxy_protocol;
|
||||
let mask_host = config
|
||||
.censorship
|
||||
.mask_host
|
||||
.clone()
|
||||
.unwrap_or_else(|| config.censorship.tls_domain.clone());
|
||||
let mask_unix_sock = config.censorship.mask_unix_sock.clone();
|
||||
let fetch_timeout = Duration::from_secs(5);
|
||||
|
||||
let cache_initial = cache.clone();
|
||||
let domains_initial = tls_domains.clone();
|
||||
let host_initial = mask_host.clone();
|
||||
let unix_sock_initial = mask_unix_sock.clone();
|
||||
let upstream_initial = upstream_manager.clone();
|
||||
tokio::spawn(async move {
|
||||
let mut join = tokio::task::JoinSet::new();
|
||||
for domain in domains_initial {
|
||||
let cache_domain = cache_initial.clone();
|
||||
let host_domain = host_initial.clone();
|
||||
let unix_sock_domain = unix_sock_initial.clone();
|
||||
let upstream_domain = upstream_initial.clone();
|
||||
join.spawn(async move {
|
||||
match crate::tls_front::fetcher::fetch_real_tls(
|
||||
&host_domain,
|
||||
port,
|
||||
&domain,
|
||||
fetch_timeout,
|
||||
Some(upstream_domain),
|
||||
proxy_protocol,
|
||||
unix_sock_domain.as_deref(),
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(res) => cache_domain.update_from_fetch(&domain, res).await,
|
||||
Err(e) => {
|
||||
warn!(domain = %domain, error = %e, "TLS emulation initial fetch failed")
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
while let Some(res) = join.join_next().await {
|
||||
if let Err(e) = res {
|
||||
warn!(error = %e, "TLS emulation initial fetch task join failed");
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
let cache_timeout = cache.clone();
|
||||
let domains_timeout = tls_domains.clone();
|
||||
let fake_cert_len = config.censorship.fake_cert_len;
|
||||
tokio::spawn(async move {
|
||||
tokio::time::sleep(fetch_timeout).await;
|
||||
for domain in domains_timeout {
|
||||
let cached = cache_timeout.get(&domain).await;
|
||||
if cached.domain == "default" {
|
||||
warn!(
|
||||
domain = %domain,
|
||||
timeout_secs = fetch_timeout.as_secs(),
|
||||
fake_cert_len,
|
||||
"TLS-front fetch not ready within timeout; using cache/default fake cert fallback"
|
||||
);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
// Periodic refresh with jitter.
|
||||
let cache_refresh = cache.clone();
|
||||
let domains_refresh = tls_domains.clone();
|
||||
let host_refresh = mask_host.clone();
|
||||
let unix_sock_refresh = mask_unix_sock.clone();
|
||||
let upstream_refresh = upstream_manager.clone();
|
||||
tokio::spawn(async move {
|
||||
loop {
|
||||
let base_secs = rand::rng().random_range(4 * 3600..=6 * 3600);
|
||||
let jitter_secs = rand::rng().random_range(0..=7200);
|
||||
tokio::time::sleep(Duration::from_secs(base_secs + jitter_secs)).await;
|
||||
|
||||
let mut join = tokio::task::JoinSet::new();
|
||||
for domain in domains_refresh.clone() {
|
||||
let cache_domain = cache_refresh.clone();
|
||||
let host_domain = host_refresh.clone();
|
||||
let unix_sock_domain = unix_sock_refresh.clone();
|
||||
let upstream_domain = upstream_refresh.clone();
|
||||
join.spawn(async move {
|
||||
match crate::tls_front::fetcher::fetch_real_tls(
|
||||
&host_domain,
|
||||
port,
|
||||
&domain,
|
||||
fetch_timeout,
|
||||
Some(upstream_domain),
|
||||
proxy_protocol,
|
||||
unix_sock_domain.as_deref(),
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(res) => cache_domain.update_from_fetch(&domain, res).await,
|
||||
Err(e) => warn!(domain = %domain, error = %e, "TLS emulation refresh failed"),
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
while let Some(res) = join.join_next().await {
|
||||
if let Err(e) = res {
|
||||
warn!(error = %e, "TLS emulation refresh task join failed");
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
Some(cache)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
let probe = run_probe(
|
||||
&config.network,
|
||||
config.general.middle_proxy_nat_stun.clone(),
|
||||
config.general.middle_proxy_nat_probe,
|
||||
config.general.stun_nat_probe_concurrency,
|
||||
)
|
||||
.await?;
|
||||
let decision = decide_network_capabilities(&config.network, &probe);
|
||||
@@ -254,6 +412,8 @@ async fn main() -> std::result::Result<(), Box<dyn std::error::Error>> {
|
||||
let prefer_ipv6 = decision.prefer_ipv6();
|
||||
let mut use_middle_proxy = config.general.use_middle_proxy && (decision.ipv4_me || decision.ipv6_me);
|
||||
let stats = Arc::new(Stats::new());
|
||||
stats.apply_telemetry_policy(TelemetryPolicy::from_config(&config.general.telemetry));
|
||||
let beobachten = Arc::new(BeobachtenStore::new());
|
||||
let rng = Arc::new(SecureRandom::new());
|
||||
|
||||
// IP Tracker initialization
|
||||
@@ -263,6 +423,12 @@ async fn main() -> std::result::Result<(), Box<dyn std::error::Error>> {
|
||||
if !config.access.user_max_unique_ips.is_empty() {
|
||||
info!("IP limits configured for {} users", config.access.user_max_unique_ips.len());
|
||||
}
|
||||
if !config.network.dns_overrides.is_empty() {
|
||||
info!(
|
||||
"Runtime DNS overrides configured: {} entries",
|
||||
config.network.dns_overrides.len()
|
||||
);
|
||||
}
|
||||
|
||||
// Connection concurrency limit
|
||||
let max_connections = Arc::new(Semaphore::new(10_000));
|
||||
@@ -277,14 +443,17 @@ async fn main() -> std::result::Result<(), Box<dyn std::error::Error>> {
|
||||
// =====================================================================
|
||||
let me_pool: Option<Arc<MePool>> = if use_middle_proxy {
|
||||
info!("=== Middle Proxy Mode ===");
|
||||
let me_nat_probe = config.general.middle_proxy_nat_probe && config.network.stun_use;
|
||||
if config.general.middle_proxy_nat_probe && !config.network.stun_use {
|
||||
info!("Middle-proxy STUN probing disabled by network.stun_use=false");
|
||||
}
|
||||
|
||||
// ad_tag (proxy_tag) for advertising
|
||||
let proxy_tag = config.general.ad_tag.as_ref().map(|tag| {
|
||||
hex::decode(tag).unwrap_or_else(|_| {
|
||||
warn!("Invalid ad_tag hex, middle proxy ad_tag will be empty");
|
||||
Vec::new()
|
||||
})
|
||||
});
|
||||
let proxy_tag = config
|
||||
.general
|
||||
.ad_tag
|
||||
.as_ref()
|
||||
.map(|tag| hex::decode(tag).expect("general.ad_tag must be validated before startup"));
|
||||
|
||||
// =============================================================
|
||||
// CRITICAL: Download Telegram proxy-secret (NOT user secret!)
|
||||
@@ -296,25 +465,30 @@ async fn main() -> std::result::Result<(), Box<dyn std::error::Error>> {
|
||||
// proxy-secret is from: https://core.telegram.org/getProxySecret
|
||||
// =============================================================
|
||||
let proxy_secret_path = config.general.proxy_secret_path.as_deref();
|
||||
match crate::transport::middle_proxy::fetch_proxy_secret(proxy_secret_path).await {
|
||||
Ok(proxy_secret) => {
|
||||
info!(
|
||||
secret_len = proxy_secret.len() as usize, // ← ЯВНЫЙ ТИП usize
|
||||
key_sig = format_args!(
|
||||
"0x{:08x}",
|
||||
if proxy_secret.len() >= 4 {
|
||||
u32::from_le_bytes([
|
||||
proxy_secret[0],
|
||||
proxy_secret[1],
|
||||
proxy_secret[2],
|
||||
proxy_secret[3],
|
||||
])
|
||||
} else {
|
||||
0
|
||||
}
|
||||
),
|
||||
"Proxy-secret loaded"
|
||||
);
|
||||
match crate::transport::middle_proxy::fetch_proxy_secret(
|
||||
proxy_secret_path,
|
||||
config.general.proxy_secret_len_max,
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(proxy_secret) => {
|
||||
info!(
|
||||
secret_len = proxy_secret.len(),
|
||||
key_sig = format_args!(
|
||||
"0x{:08x}",
|
||||
if proxy_secret.len() >= 4 {
|
||||
u32::from_le_bytes([
|
||||
proxy_secret[0],
|
||||
proxy_secret[1],
|
||||
proxy_secret[2],
|
||||
proxy_secret[3],
|
||||
])
|
||||
} else {
|
||||
0
|
||||
}
|
||||
),
|
||||
"Proxy-secret loaded"
|
||||
);
|
||||
|
||||
// Load ME config (v4/v6) + default DC
|
||||
let mut cfg_v4 = fetch_proxy_config(
|
||||
@@ -339,9 +513,10 @@ match crate::transport::middle_proxy::fetch_proxy_secret(proxy_secret_path).awai
|
||||
proxy_tag,
|
||||
proxy_secret,
|
||||
config.general.middle_proxy_nat_ip,
|
||||
config.general.middle_proxy_nat_probe,
|
||||
config.general.middle_proxy_nat_stun.clone(),
|
||||
config.general.middle_proxy_nat_stun_servers.clone(),
|
||||
me_nat_probe,
|
||||
None,
|
||||
config.network.stun_servers.clone(),
|
||||
config.general.stun_nat_probe_concurrency,
|
||||
probe.detected_ipv6,
|
||||
config.timeouts.me_one_retry,
|
||||
config.timeouts.me_one_timeout_ms,
|
||||
@@ -349,6 +524,7 @@ match crate::transport::middle_proxy::fetch_proxy_secret(proxy_secret_path).awai
|
||||
cfg_v6.map.clone(),
|
||||
cfg_v4.default_dc.or(cfg_v6.default_dc),
|
||||
decision.clone(),
|
||||
Some(upstream_manager.clone()),
|
||||
rng.clone(),
|
||||
stats.clone(),
|
||||
config.general.me_keepalive_enabled,
|
||||
@@ -366,41 +542,44 @@ match crate::transport::middle_proxy::fetch_proxy_secret(proxy_secret_path).awai
|
||||
config.general.me_pool_drain_ttl_secs,
|
||||
config.general.effective_me_pool_force_close_secs(),
|
||||
config.general.me_pool_min_fresh_ratio,
|
||||
config.general.me_hardswap_warmup_delay_min_ms,
|
||||
config.general.me_hardswap_warmup_delay_max_ms,
|
||||
config.general.me_hardswap_warmup_extra_passes,
|
||||
config.general.me_hardswap_warmup_pass_backoff_base_ms,
|
||||
config.general.me_socks_kdf_policy,
|
||||
config.general.me_route_backpressure_base_timeout_ms,
|
||||
config.general.me_route_backpressure_high_timeout_ms,
|
||||
config.general.me_route_backpressure_high_watermark_pct,
|
||||
);
|
||||
|
||||
let pool_size = config.general.middle_proxy_pool_size.max(1);
|
||||
match pool.init(pool_size, &rng).await {
|
||||
Ok(()) => {
|
||||
info!("Middle-End pool initialized successfully");
|
||||
loop {
|
||||
match pool.init(pool_size, &rng).await {
|
||||
Ok(()) => {
|
||||
info!("Middle-End pool initialized successfully");
|
||||
|
||||
// Phase 4: Start health monitor
|
||||
let pool_clone = pool.clone();
|
||||
let rng_clone = rng.clone();
|
||||
let min_conns = pool_size;
|
||||
tokio::spawn(async move {
|
||||
crate::transport::middle_proxy::me_health_monitor(
|
||||
pool_clone, rng_clone, min_conns,
|
||||
)
|
||||
.await;
|
||||
});
|
||||
// Phase 4: Start health monitor
|
||||
let pool_clone = pool.clone();
|
||||
let rng_clone = rng.clone();
|
||||
let min_conns = pool_size;
|
||||
tokio::spawn(async move {
|
||||
crate::transport::middle_proxy::me_health_monitor(
|
||||
pool_clone, rng_clone, min_conns,
|
||||
)
|
||||
.await;
|
||||
});
|
||||
|
||||
// Periodic ME connection rotation
|
||||
let pool_clone_rot = pool.clone();
|
||||
let rng_clone_rot = rng.clone();
|
||||
tokio::spawn(async move {
|
||||
crate::transport::middle_proxy::me_rotation_task(
|
||||
pool_clone_rot,
|
||||
rng_clone_rot,
|
||||
std::time::Duration::from_secs(1800),
|
||||
)
|
||||
.await;
|
||||
});
|
||||
|
||||
Some(pool)
|
||||
}
|
||||
Err(e) => {
|
||||
error!(error = %e, "Failed to initialize ME pool. Falling back to direct mode.");
|
||||
None
|
||||
break Some(pool);
|
||||
}
|
||||
Err(e) => {
|
||||
warn!(
|
||||
error = %e,
|
||||
retry_in_secs = 2,
|
||||
"ME pool is not ready yet; retrying startup initialization"
|
||||
);
|
||||
pool.reset_stun_state();
|
||||
tokio::time::sleep(Duration::from_secs(2)).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -417,6 +596,7 @@ match crate::transport::middle_proxy::fetch_proxy_secret(proxy_secret_path).awai
|
||||
if me_pool.is_some() {
|
||||
info!("Transport: Middle-End Proxy - all DC-over-RPC");
|
||||
} else {
|
||||
let _ = use_middle_proxy;
|
||||
use_middle_proxy = false;
|
||||
// Make runtime config reflect direct-only mode for handlers.
|
||||
config.general.use_middle_proxy = false;
|
||||
@@ -431,77 +611,8 @@ match crate::transport::middle_proxy::fetch_proxy_secret(proxy_secret_path).awai
|
||||
Duration::from_secs(config.access.replay_window_secs),
|
||||
));
|
||||
|
||||
let upstream_manager = Arc::new(UpstreamManager::new(config.upstreams.clone()));
|
||||
let buffer_pool = Arc::new(BufferPool::with_config(16 * 1024, 4096));
|
||||
|
||||
// TLS front cache (optional emulation)
|
||||
let mut tls_domains = Vec::with_capacity(1 + config.censorship.tls_domains.len());
|
||||
tls_domains.push(config.censorship.tls_domain.clone());
|
||||
for d in &config.censorship.tls_domains {
|
||||
if !tls_domains.contains(d) {
|
||||
tls_domains.push(d.clone());
|
||||
}
|
||||
}
|
||||
|
||||
let tls_cache: Option<Arc<TlsFrontCache>> = if config.censorship.tls_emulation {
|
||||
let cache = Arc::new(TlsFrontCache::new(
|
||||
&tls_domains,
|
||||
config.censorship.fake_cert_len,
|
||||
&config.censorship.tls_front_dir,
|
||||
));
|
||||
|
||||
cache.load_from_disk().await;
|
||||
|
||||
let port = config.censorship.mask_port;
|
||||
let mask_host = config.censorship.mask_host.clone()
|
||||
.unwrap_or_else(|| config.censorship.tls_domain.clone());
|
||||
// Initial synchronous fetch to warm cache before serving clients.
|
||||
for domain in tls_domains.clone() {
|
||||
match crate::tls_front::fetcher::fetch_real_tls(
|
||||
&mask_host,
|
||||
port,
|
||||
&domain,
|
||||
Duration::from_secs(5),
|
||||
Some(upstream_manager.clone()),
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(res) => cache.update_from_fetch(&domain, res).await,
|
||||
Err(e) => warn!(domain = %domain, error = %e, "TLS emulation fetch failed"),
|
||||
}
|
||||
}
|
||||
|
||||
// Periodic refresh with jitter.
|
||||
let cache_clone = cache.clone();
|
||||
let domains = tls_domains.clone();
|
||||
let upstream_for_task = upstream_manager.clone();
|
||||
tokio::spawn(async move {
|
||||
loop {
|
||||
let base_secs = rand::rng().random_range(4 * 3600..=6 * 3600);
|
||||
let jitter_secs = rand::rng().random_range(0..=7200);
|
||||
tokio::time::sleep(Duration::from_secs(base_secs + jitter_secs)).await;
|
||||
for domain in &domains {
|
||||
match crate::tls_front::fetcher::fetch_real_tls(
|
||||
&mask_host,
|
||||
port,
|
||||
domain,
|
||||
Duration::from_secs(5),
|
||||
Some(upstream_for_task.clone()),
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(res) => cache_clone.update_from_fetch(domain, res).await,
|
||||
Err(e) => warn!(domain = %domain, error = %e, "TLS emulation refresh failed"),
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
Some(cache)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
// Middle-End ping before DC connectivity
|
||||
if let Some(ref pool) = me_pool {
|
||||
let me_results = run_me_ping(pool, &rng).await;
|
||||
@@ -525,7 +636,15 @@ match crate::transport::middle_proxy::fetch_proxy_secret(proxy_secret_path).awai
|
||||
} else {
|
||||
info!(" No ME connectivity");
|
||||
}
|
||||
info!(" via direct");
|
||||
let me_route = format_me_route(
|
||||
&config.upstreams,
|
||||
&me_results,
|
||||
prefer_ipv6,
|
||||
v4_ok,
|
||||
v6_ok,
|
||||
)
|
||||
.await;
|
||||
info!(" via {}", me_route);
|
||||
info!("============================================================");
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
@@ -594,14 +713,12 @@ match crate::transport::middle_proxy::fetch_proxy_secret(proxy_secret_path).awai
|
||||
} else {
|
||||
info!(" IPv4 in use / IPv6 is fallback");
|
||||
}
|
||||
} else {
|
||||
if v6_works && !v4_works {
|
||||
info!(" IPv6 only / IPv4 unavailable)");
|
||||
} else if v4_works && !v6_works {
|
||||
info!(" IPv4 only / IPv6 unavailable)");
|
||||
} else if !v6_works && !v4_works {
|
||||
info!(" No DC connectivity");
|
||||
}
|
||||
} else if v6_works && !v4_works {
|
||||
info!(" IPv6 only / IPv4 unavailable");
|
||||
} else if v4_works && !v6_works {
|
||||
info!(" IPv4 only / IPv6 unavailable");
|
||||
} else if !v6_works && !v4_works {
|
||||
info!(" No DC connectivity");
|
||||
}
|
||||
|
||||
info!(" via {}", upstream_result.upstream_name);
|
||||
@@ -668,14 +785,8 @@ match crate::transport::middle_proxy::fetch_proxy_secret(proxy_secret_path).awai
|
||||
rc_clone.run_periodic_cleanup().await;
|
||||
});
|
||||
|
||||
let detected_ip_v4: Option<std::net::IpAddr> = probe
|
||||
.reflected_ipv4
|
||||
.map(|s| s.ip())
|
||||
.or_else(|| probe.detected_ipv4.map(std::net::IpAddr::V4));
|
||||
let detected_ip_v6: Option<std::net::IpAddr> = probe
|
||||
.reflected_ipv6
|
||||
.map(|s| s.ip())
|
||||
.or_else(|| probe.detected_ipv6.map(std::net::IpAddr::V6));
|
||||
let detected_ip_v4: Option<std::net::IpAddr> = probe.detected_ipv4.map(std::net::IpAddr::V4);
|
||||
let detected_ip_v6: Option<std::net::IpAddr> = probe.detected_ipv6.map(std::net::IpAddr::V6);
|
||||
debug!(
|
||||
"Detected IPs: v4={:?} v6={:?}",
|
||||
detected_ip_v4, detected_ip_v6
|
||||
@@ -694,6 +805,47 @@ match crate::transport::middle_proxy::fetch_proxy_secret(proxy_secret_path).awai
|
||||
detected_ip_v6,
|
||||
);
|
||||
|
||||
let stats_policy = stats.clone();
|
||||
let mut config_rx_policy = config_rx.clone();
|
||||
let me_pool_policy = me_pool.clone();
|
||||
tokio::spawn(async move {
|
||||
loop {
|
||||
if config_rx_policy.changed().await.is_err() {
|
||||
break;
|
||||
}
|
||||
let cfg = config_rx_policy.borrow_and_update().clone();
|
||||
stats_policy.apply_telemetry_policy(TelemetryPolicy::from_config(&cfg.general.telemetry));
|
||||
if let Some(pool) = &me_pool_policy {
|
||||
pool.update_runtime_transport_policy(
|
||||
cfg.general.me_socks_kdf_policy,
|
||||
cfg.general.me_route_backpressure_base_timeout_ms,
|
||||
cfg.general.me_route_backpressure_high_timeout_ms,
|
||||
cfg.general.me_route_backpressure_high_watermark_pct,
|
||||
);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
let beobachten_writer = beobachten.clone();
|
||||
let config_rx_beobachten = config_rx.clone();
|
||||
tokio::spawn(async move {
|
||||
loop {
|
||||
let cfg = config_rx_beobachten.borrow().clone();
|
||||
let sleep_secs = cfg.general.beobachten_flush_secs.max(1);
|
||||
|
||||
if cfg.general.beobachten {
|
||||
let ttl = Duration::from_secs(cfg.general.beobachten_minutes.saturating_mul(60));
|
||||
let path = cfg.general.beobachten_file.clone();
|
||||
let snapshot = beobachten_writer.snapshot_text(ttl);
|
||||
if let Err(e) = write_beobachten_snapshot(&path, &snapshot).await {
|
||||
warn!(error = %e, path = %path, "Failed to flush beobachten snapshot");
|
||||
}
|
||||
}
|
||||
|
||||
tokio::time::sleep(Duration::from_secs(sleep_secs)).await;
|
||||
}
|
||||
});
|
||||
|
||||
if let Some(ref pool) = me_pool {
|
||||
let pool_clone = pool.clone();
|
||||
let rng_clone = rng.clone();
|
||||
@@ -706,6 +858,18 @@ match crate::transport::middle_proxy::fetch_proxy_secret(proxy_secret_path).awai
|
||||
)
|
||||
.await;
|
||||
});
|
||||
|
||||
let pool_clone_rot = pool.clone();
|
||||
let rng_clone_rot = rng.clone();
|
||||
let config_rx_clone_rot = config_rx.clone();
|
||||
tokio::spawn(async move {
|
||||
crate::transport::middle_proxy::me_rotation_task(
|
||||
pool_clone_rot,
|
||||
rng_clone_rot,
|
||||
config_rx_clone_rot,
|
||||
)
|
||||
.await;
|
||||
});
|
||||
}
|
||||
|
||||
let mut listeners = Vec::new();
|
||||
@@ -850,6 +1014,7 @@ match crate::transport::middle_proxy::fetch_proxy_secret(proxy_secret_path).awai
|
||||
let me_pool = me_pool.clone();
|
||||
let tls_cache = tls_cache.clone();
|
||||
let ip_tracker = ip_tracker.clone();
|
||||
let beobachten = beobachten.clone();
|
||||
let max_connections_unix = max_connections.clone();
|
||||
|
||||
tokio::spawn(async move {
|
||||
@@ -877,6 +1042,7 @@ match crate::transport::middle_proxy::fetch_proxy_secret(proxy_secret_path).awai
|
||||
let me_pool = me_pool.clone();
|
||||
let tls_cache = tls_cache.clone();
|
||||
let ip_tracker = ip_tracker.clone();
|
||||
let beobachten = beobachten.clone();
|
||||
let proxy_protocol_enabled = config.server.proxy_protocol;
|
||||
|
||||
tokio::spawn(async move {
|
||||
@@ -884,7 +1050,7 @@ match crate::transport::middle_proxy::fetch_proxy_secret(proxy_secret_path).awai
|
||||
if let Err(e) = crate::proxy::client::handle_client_stream(
|
||||
stream, fake_peer, config, stats,
|
||||
upstream_manager, replay_checker, buffer_pool, rng,
|
||||
me_pool, tls_cache, ip_tracker, proxy_protocol_enabled,
|
||||
me_pool, tls_cache, ip_tracker, beobachten, proxy_protocol_enabled,
|
||||
).await {
|
||||
debug!(error = %e, "Unix socket connection error");
|
||||
}
|
||||
@@ -932,9 +1098,20 @@ match crate::transport::middle_proxy::fetch_proxy_secret(proxy_secret_path).awai
|
||||
|
||||
if let Some(port) = config.server.metrics_port {
|
||||
let stats = stats.clone();
|
||||
let beobachten = beobachten.clone();
|
||||
let config_rx_metrics = config_rx.clone();
|
||||
let ip_tracker_metrics = ip_tracker.clone();
|
||||
let whitelist = config.server.metrics_whitelist.clone();
|
||||
tokio::spawn(async move {
|
||||
metrics::serve(port, stats, whitelist).await;
|
||||
metrics::serve(
|
||||
port,
|
||||
stats,
|
||||
beobachten,
|
||||
ip_tracker_metrics,
|
||||
config_rx_metrics,
|
||||
whitelist,
|
||||
)
|
||||
.await;
|
||||
});
|
||||
}
|
||||
|
||||
@@ -948,6 +1125,7 @@ match crate::transport::middle_proxy::fetch_proxy_secret(proxy_secret_path).awai
|
||||
let me_pool = me_pool.clone();
|
||||
let tls_cache = tls_cache.clone();
|
||||
let ip_tracker = ip_tracker.clone();
|
||||
let beobachten = beobachten.clone();
|
||||
let max_connections_tcp = max_connections.clone();
|
||||
|
||||
tokio::spawn(async move {
|
||||
@@ -970,6 +1148,7 @@ match crate::transport::middle_proxy::fetch_proxy_secret(proxy_secret_path).awai
|
||||
let me_pool = me_pool.clone();
|
||||
let tls_cache = tls_cache.clone();
|
||||
let ip_tracker = ip_tracker.clone();
|
||||
let beobachten = beobachten.clone();
|
||||
let proxy_protocol_enabled = listener_proxy_protocol;
|
||||
|
||||
tokio::spawn(async move {
|
||||
@@ -986,6 +1165,7 @@ match crate::transport::middle_proxy::fetch_proxy_secret(proxy_secret_path).awai
|
||||
me_pool,
|
||||
tls_cache,
|
||||
ip_tracker,
|
||||
beobachten,
|
||||
proxy_protocol_enabled,
|
||||
)
|
||||
.run()
|
||||
|
||||
659
src/metrics.rs
659
src/metrics.rs
@@ -1,8 +1,10 @@
|
||||
use std::convert::Infallible;
|
||||
use std::collections::{BTreeSet, HashMap};
|
||||
use std::net::SocketAddr;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
use http_body_util::{Full, BodyExt};
|
||||
use http_body_util::Full;
|
||||
use hyper::body::Bytes;
|
||||
use hyper::server::conn::http1;
|
||||
use hyper::service::service_fn;
|
||||
@@ -11,9 +13,19 @@ use ipnetwork::IpNetwork;
|
||||
use tokio::net::TcpListener;
|
||||
use tracing::{info, warn, debug};
|
||||
|
||||
use crate::config::ProxyConfig;
|
||||
use crate::ip_tracker::UserIpTracker;
|
||||
use crate::stats::beobachten::BeobachtenStore;
|
||||
use crate::stats::Stats;
|
||||
|
||||
pub async fn serve(port: u16, stats: Arc<Stats>, whitelist: Vec<IpNetwork>) {
|
||||
pub async fn serve(
|
||||
port: u16,
|
||||
stats: Arc<Stats>,
|
||||
beobachten: Arc<BeobachtenStore>,
|
||||
ip_tracker: Arc<UserIpTracker>,
|
||||
config_rx: tokio::sync::watch::Receiver<Arc<ProxyConfig>>,
|
||||
whitelist: Vec<IpNetwork>,
|
||||
) {
|
||||
let addr = SocketAddr::from(([0, 0, 0, 0], port));
|
||||
let listener = match TcpListener::bind(addr).await {
|
||||
Ok(l) => l,
|
||||
@@ -22,7 +34,7 @@ pub async fn serve(port: u16, stats: Arc<Stats>, whitelist: Vec<IpNetwork>) {
|
||||
return;
|
||||
}
|
||||
};
|
||||
info!("Metrics endpoint: http://{}/metrics", addr);
|
||||
info!("Metrics endpoint: http://{}/metrics and /beobachten", addr);
|
||||
|
||||
loop {
|
||||
let (stream, peer) = match listener.accept().await {
|
||||
@@ -39,10 +51,16 @@ pub async fn serve(port: u16, stats: Arc<Stats>, whitelist: Vec<IpNetwork>) {
|
||||
}
|
||||
|
||||
let stats = stats.clone();
|
||||
let beobachten = beobachten.clone();
|
||||
let ip_tracker = ip_tracker.clone();
|
||||
let config_rx_conn = config_rx.clone();
|
||||
tokio::spawn(async move {
|
||||
let svc = service_fn(move |req| {
|
||||
let stats = stats.clone();
|
||||
async move { handle(req, &stats) }
|
||||
let beobachten = beobachten.clone();
|
||||
let ip_tracker = ip_tracker.clone();
|
||||
let config = config_rx_conn.borrow().clone();
|
||||
async move { handle(req, &stats, &beobachten, &ip_tracker, &config).await }
|
||||
});
|
||||
if let Err(e) = http1::Builder::new()
|
||||
.serve_connection(hyper_util::rt::TokioIo::new(stream), svc)
|
||||
@@ -54,141 +72,440 @@ pub async fn serve(port: u16, stats: Arc<Stats>, whitelist: Vec<IpNetwork>) {
|
||||
}
|
||||
}
|
||||
|
||||
fn handle<B>(req: Request<B>, stats: &Stats) -> Result<Response<Full<Bytes>>, Infallible> {
|
||||
if req.uri().path() != "/metrics" {
|
||||
async fn handle<B>(
|
||||
req: Request<B>,
|
||||
stats: &Stats,
|
||||
beobachten: &BeobachtenStore,
|
||||
ip_tracker: &UserIpTracker,
|
||||
config: &ProxyConfig,
|
||||
) -> Result<Response<Full<Bytes>>, Infallible> {
|
||||
if req.uri().path() == "/metrics" {
|
||||
let body = render_metrics(stats, config, ip_tracker).await;
|
||||
let resp = Response::builder()
|
||||
.status(StatusCode::NOT_FOUND)
|
||||
.body(Full::new(Bytes::from("Not Found\n")))
|
||||
.status(StatusCode::OK)
|
||||
.header("content-type", "text/plain; version=0.0.4; charset=utf-8")
|
||||
.body(Full::new(Bytes::from(body)))
|
||||
.unwrap();
|
||||
return Ok(resp);
|
||||
}
|
||||
|
||||
if req.uri().path() == "/beobachten" {
|
||||
let body = render_beobachten(beobachten, config);
|
||||
let resp = Response::builder()
|
||||
.status(StatusCode::OK)
|
||||
.header("content-type", "text/plain; charset=utf-8")
|
||||
.body(Full::new(Bytes::from(body)))
|
||||
.unwrap();
|
||||
return Ok(resp);
|
||||
}
|
||||
|
||||
let body = render_metrics(stats);
|
||||
let resp = Response::builder()
|
||||
.status(StatusCode::OK)
|
||||
.header("content-type", "text/plain; version=0.0.4; charset=utf-8")
|
||||
.body(Full::new(Bytes::from(body)))
|
||||
.status(StatusCode::NOT_FOUND)
|
||||
.body(Full::new(Bytes::from("Not Found\n")))
|
||||
.unwrap();
|
||||
Ok(resp)
|
||||
}
|
||||
|
||||
fn render_metrics(stats: &Stats) -> String {
|
||||
fn render_beobachten(beobachten: &BeobachtenStore, config: &ProxyConfig) -> String {
|
||||
if !config.general.beobachten {
|
||||
return "beobachten disabled\n".to_string();
|
||||
}
|
||||
|
||||
let ttl = Duration::from_secs(config.general.beobachten_minutes.saturating_mul(60));
|
||||
beobachten.snapshot_text(ttl)
|
||||
}
|
||||
|
||||
async fn render_metrics(stats: &Stats, config: &ProxyConfig, ip_tracker: &UserIpTracker) -> String {
|
||||
use std::fmt::Write;
|
||||
let mut out = String::with_capacity(4096);
|
||||
let telemetry = stats.telemetry_policy();
|
||||
let core_enabled = telemetry.core_enabled;
|
||||
let user_enabled = telemetry.user_enabled;
|
||||
let me_allows_normal = telemetry.me_level.allows_normal();
|
||||
let me_allows_debug = telemetry.me_level.allows_debug();
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_uptime_seconds Proxy uptime");
|
||||
let _ = writeln!(out, "# TYPE telemt_uptime_seconds gauge");
|
||||
let _ = writeln!(out, "telemt_uptime_seconds {:.1}", stats.uptime_secs());
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_telemetry_core_enabled Runtime core telemetry switch");
|
||||
let _ = writeln!(out, "# TYPE telemt_telemetry_core_enabled gauge");
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_telemetry_core_enabled {}",
|
||||
if core_enabled { 1 } else { 0 }
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_telemetry_user_enabled Runtime per-user telemetry switch");
|
||||
let _ = writeln!(out, "# TYPE telemt_telemetry_user_enabled gauge");
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_telemetry_user_enabled {}",
|
||||
if user_enabled { 1 } else { 0 }
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_telemetry_me_level Runtime ME telemetry level flag");
|
||||
let _ = writeln!(out, "# TYPE telemt_telemetry_me_level gauge");
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_telemetry_me_level{{level=\"silent\"}} {}",
|
||||
if matches!(telemetry.me_level, crate::config::MeTelemetryLevel::Silent) {
|
||||
1
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_telemetry_me_level{{level=\"normal\"}} {}",
|
||||
if matches!(telemetry.me_level, crate::config::MeTelemetryLevel::Normal) {
|
||||
1
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_telemetry_me_level{{level=\"debug\"}} {}",
|
||||
if matches!(telemetry.me_level, crate::config::MeTelemetryLevel::Debug) {
|
||||
1
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_connections_total Total accepted connections");
|
||||
let _ = writeln!(out, "# TYPE telemt_connections_total counter");
|
||||
let _ = writeln!(out, "telemt_connections_total {}", stats.get_connects_all());
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_connections_total {}",
|
||||
if core_enabled { stats.get_connects_all() } else { 0 }
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_connections_bad_total Bad/rejected connections");
|
||||
let _ = writeln!(out, "# TYPE telemt_connections_bad_total counter");
|
||||
let _ = writeln!(out, "telemt_connections_bad_total {}", stats.get_connects_bad());
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_connections_bad_total {}",
|
||||
if core_enabled { stats.get_connects_bad() } else { 0 }
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_handshake_timeouts_total Handshake timeouts");
|
||||
let _ = writeln!(out, "# TYPE telemt_handshake_timeouts_total counter");
|
||||
let _ = writeln!(out, "telemt_handshake_timeouts_total {}", stats.get_handshake_timeouts());
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_handshake_timeouts_total {}",
|
||||
if core_enabled {
|
||||
stats.get_handshake_timeouts()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_me_keepalive_sent_total ME keepalive frames sent");
|
||||
let _ = writeln!(out, "# TYPE telemt_me_keepalive_sent_total counter");
|
||||
let _ = writeln!(out, "telemt_me_keepalive_sent_total {}", stats.get_me_keepalive_sent());
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_me_keepalive_sent_total {}",
|
||||
if me_allows_debug {
|
||||
stats.get_me_keepalive_sent()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_me_keepalive_failed_total ME keepalive send failures");
|
||||
let _ = writeln!(out, "# TYPE telemt_me_keepalive_failed_total counter");
|
||||
let _ = writeln!(out, "telemt_me_keepalive_failed_total {}", stats.get_me_keepalive_failed());
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_me_keepalive_failed_total {}",
|
||||
if me_allows_normal {
|
||||
stats.get_me_keepalive_failed()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_me_keepalive_pong_total ME keepalive pong replies");
|
||||
let _ = writeln!(out, "# TYPE telemt_me_keepalive_pong_total counter");
|
||||
let _ = writeln!(out, "telemt_me_keepalive_pong_total {}", stats.get_me_keepalive_pong());
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_me_keepalive_pong_total {}",
|
||||
if me_allows_debug {
|
||||
stats.get_me_keepalive_pong()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_me_keepalive_timeout_total ME keepalive ping timeouts");
|
||||
let _ = writeln!(out, "# TYPE telemt_me_keepalive_timeout_total counter");
|
||||
let _ = writeln!(out, "telemt_me_keepalive_timeout_total {}", stats.get_me_keepalive_timeout());
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_me_keepalive_timeout_total {}",
|
||||
if me_allows_normal {
|
||||
stats.get_me_keepalive_timeout()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_me_reconnect_attempts_total ME reconnect attempts");
|
||||
let _ = writeln!(out, "# TYPE telemt_me_reconnect_attempts_total counter");
|
||||
let _ = writeln!(out, "telemt_me_reconnect_attempts_total {}", stats.get_me_reconnect_attempts());
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_me_reconnect_attempts_total {}",
|
||||
if me_allows_normal {
|
||||
stats.get_me_reconnect_attempts()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_me_reconnect_success_total ME reconnect successes");
|
||||
let _ = writeln!(out, "# TYPE telemt_me_reconnect_success_total counter");
|
||||
let _ = writeln!(out, "telemt_me_reconnect_success_total {}", stats.get_me_reconnect_success());
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_me_reconnect_success_total {}",
|
||||
if me_allows_normal {
|
||||
stats.get_me_reconnect_success()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_me_crc_mismatch_total ME CRC mismatches");
|
||||
let _ = writeln!(out, "# TYPE telemt_me_crc_mismatch_total counter");
|
||||
let _ = writeln!(out, "telemt_me_crc_mismatch_total {}", stats.get_me_crc_mismatch());
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_me_crc_mismatch_total {}",
|
||||
if me_allows_normal {
|
||||
stats.get_me_crc_mismatch()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_me_seq_mismatch_total ME sequence mismatches");
|
||||
let _ = writeln!(out, "# TYPE telemt_me_seq_mismatch_total counter");
|
||||
let _ = writeln!(out, "telemt_me_seq_mismatch_total {}", stats.get_me_seq_mismatch());
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_me_seq_mismatch_total {}",
|
||||
if me_allows_normal {
|
||||
stats.get_me_seq_mismatch()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_me_route_drop_no_conn_total ME route drops: no conn");
|
||||
let _ = writeln!(out, "# TYPE telemt_me_route_drop_no_conn_total counter");
|
||||
let _ = writeln!(out, "telemt_me_route_drop_no_conn_total {}", stats.get_me_route_drop_no_conn());
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_me_route_drop_no_conn_total {}",
|
||||
if me_allows_normal {
|
||||
stats.get_me_route_drop_no_conn()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_me_route_drop_channel_closed_total ME route drops: channel closed");
|
||||
let _ = writeln!(out, "# TYPE telemt_me_route_drop_channel_closed_total counter");
|
||||
let _ = writeln!(out, "telemt_me_route_drop_channel_closed_total {}", stats.get_me_route_drop_channel_closed());
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_me_route_drop_channel_closed_total {}",
|
||||
if me_allows_normal {
|
||||
stats.get_me_route_drop_channel_closed()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_me_route_drop_queue_full_total ME route drops: queue full");
|
||||
let _ = writeln!(out, "# TYPE telemt_me_route_drop_queue_full_total counter");
|
||||
let _ = writeln!(out, "telemt_me_route_drop_queue_full_total {}", stats.get_me_route_drop_queue_full());
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_me_route_drop_queue_full_total {}",
|
||||
if me_allows_normal {
|
||||
stats.get_me_route_drop_queue_full()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"# HELP telemt_me_route_drop_queue_full_profile_total ME route drops: queue full by adaptive profile"
|
||||
);
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"# TYPE telemt_me_route_drop_queue_full_profile_total counter"
|
||||
);
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_me_route_drop_queue_full_profile_total{{profile=\"base\"}} {}",
|
||||
if me_allows_normal {
|
||||
stats.get_me_route_drop_queue_full_base()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_me_route_drop_queue_full_profile_total{{profile=\"high\"}} {}",
|
||||
if me_allows_normal {
|
||||
stats.get_me_route_drop_queue_full_high()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"# HELP telemt_me_socks_kdf_policy_total SOCKS KDF policy outcomes"
|
||||
);
|
||||
let _ = writeln!(out, "# TYPE telemt_me_socks_kdf_policy_total counter");
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_me_socks_kdf_policy_total{{policy=\"strict\",outcome=\"reject\"}} {}",
|
||||
if me_allows_normal {
|
||||
stats.get_me_socks_kdf_strict_reject()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_me_socks_kdf_policy_total{{policy=\"compat\",outcome=\"fallback\"}} {}",
|
||||
if me_allows_debug {
|
||||
stats.get_me_socks_kdf_compat_fallback()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_secure_padding_invalid_total Invalid secure frame lengths");
|
||||
let _ = writeln!(out, "# TYPE telemt_secure_padding_invalid_total counter");
|
||||
let _ = writeln!(out, "telemt_secure_padding_invalid_total {}", stats.get_secure_padding_invalid());
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_secure_padding_invalid_total {}",
|
||||
if me_allows_normal {
|
||||
stats.get_secure_padding_invalid()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_desync_total Total crypto-desync detections");
|
||||
let _ = writeln!(out, "# TYPE telemt_desync_total counter");
|
||||
let _ = writeln!(out, "telemt_desync_total {}", stats.get_desync_total());
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_desync_total {}",
|
||||
if me_allows_normal {
|
||||
stats.get_desync_total()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_desync_full_logged_total Full forensic desync logs emitted");
|
||||
let _ = writeln!(out, "# TYPE telemt_desync_full_logged_total counter");
|
||||
let _ = writeln!(out, "telemt_desync_full_logged_total {}", stats.get_desync_full_logged());
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_desync_full_logged_total {}",
|
||||
if me_allows_normal {
|
||||
stats.get_desync_full_logged()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_desync_suppressed_total Suppressed desync forensic events");
|
||||
let _ = writeln!(out, "# TYPE telemt_desync_suppressed_total counter");
|
||||
let _ = writeln!(out, "telemt_desync_suppressed_total {}", stats.get_desync_suppressed());
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_desync_suppressed_total {}",
|
||||
if me_allows_normal {
|
||||
stats.get_desync_suppressed()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_desync_frames_bucket_total Desync count by frames_ok bucket");
|
||||
let _ = writeln!(out, "# TYPE telemt_desync_frames_bucket_total counter");
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_desync_frames_bucket_total{{bucket=\"0\"}} {}",
|
||||
stats.get_desync_frames_bucket_0()
|
||||
if me_allows_normal {
|
||||
stats.get_desync_frames_bucket_0()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_desync_frames_bucket_total{{bucket=\"1_2\"}} {}",
|
||||
stats.get_desync_frames_bucket_1_2()
|
||||
if me_allows_normal {
|
||||
stats.get_desync_frames_bucket_1_2()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_desync_frames_bucket_total{{bucket=\"3_10\"}} {}",
|
||||
stats.get_desync_frames_bucket_3_10()
|
||||
if me_allows_normal {
|
||||
stats.get_desync_frames_bucket_3_10()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_desync_frames_bucket_total{{bucket=\"gt_10\"}} {}",
|
||||
stats.get_desync_frames_bucket_gt_10()
|
||||
if me_allows_normal {
|
||||
stats.get_desync_frames_bucket_gt_10()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_pool_swap_total Successful ME pool swaps");
|
||||
let _ = writeln!(out, "# TYPE telemt_pool_swap_total counter");
|
||||
let _ = writeln!(out, "telemt_pool_swap_total {}", stats.get_pool_swap_total());
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_pool_swap_total {}",
|
||||
if me_allows_debug {
|
||||
stats.get_pool_swap_total()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_pool_drain_active Active draining ME writers");
|
||||
let _ = writeln!(out, "# TYPE telemt_pool_drain_active gauge");
|
||||
let _ = writeln!(out, "telemt_pool_drain_active {}", stats.get_pool_drain_active());
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_pool_drain_active {}",
|
||||
if me_allows_debug {
|
||||
stats.get_pool_drain_active()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_pool_force_close_total Forced close events for draining writers");
|
||||
let _ = writeln!(out, "# TYPE telemt_pool_force_close_total counter");
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_pool_force_close_total {}",
|
||||
stats.get_pool_force_close_total()
|
||||
if me_allows_normal {
|
||||
stats.get_pool_force_close_total()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_pool_stale_pick_total Stale writer fallback picks for new binds");
|
||||
@@ -196,7 +513,132 @@ fn render_metrics(stats: &Stats) -> String {
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_pool_stale_pick_total {}",
|
||||
stats.get_pool_stale_pick_total()
|
||||
if me_allows_normal {
|
||||
stats.get_pool_stale_pick_total()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_me_writer_removed_total Total ME writer removals");
|
||||
let _ = writeln!(out, "# TYPE telemt_me_writer_removed_total counter");
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_me_writer_removed_total {}",
|
||||
if me_allows_debug {
|
||||
stats.get_me_writer_removed_total()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"# HELP telemt_me_writer_removed_unexpected_total Unexpected ME writer removals that triggered refill"
|
||||
);
|
||||
let _ = writeln!(out, "# TYPE telemt_me_writer_removed_unexpected_total counter");
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_me_writer_removed_unexpected_total {}",
|
||||
if me_allows_normal {
|
||||
stats.get_me_writer_removed_unexpected_total()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_me_refill_triggered_total Immediate ME refill runs started");
|
||||
let _ = writeln!(out, "# TYPE telemt_me_refill_triggered_total counter");
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_me_refill_triggered_total {}",
|
||||
if me_allows_debug {
|
||||
stats.get_me_refill_triggered_total()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"# HELP telemt_me_refill_skipped_inflight_total Immediate ME refill skips due to inflight dedup"
|
||||
);
|
||||
let _ = writeln!(out, "# TYPE telemt_me_refill_skipped_inflight_total counter");
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_me_refill_skipped_inflight_total {}",
|
||||
if me_allows_debug {
|
||||
stats.get_me_refill_skipped_inflight_total()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_me_refill_failed_total Immediate ME refill failures");
|
||||
let _ = writeln!(out, "# TYPE telemt_me_refill_failed_total counter");
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_me_refill_failed_total {}",
|
||||
if me_allows_normal {
|
||||
stats.get_me_refill_failed_total()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"# HELP telemt_me_writer_restored_same_endpoint_total Refilled ME writer restored on the same endpoint"
|
||||
);
|
||||
let _ = writeln!(out, "# TYPE telemt_me_writer_restored_same_endpoint_total counter");
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_me_writer_restored_same_endpoint_total {}",
|
||||
if me_allows_normal {
|
||||
stats.get_me_writer_restored_same_endpoint_total()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"# HELP telemt_me_writer_restored_fallback_total Refilled ME writer restored via fallback endpoint"
|
||||
);
|
||||
let _ = writeln!(out, "# TYPE telemt_me_writer_restored_fallback_total counter");
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_me_writer_restored_fallback_total {}",
|
||||
if me_allows_normal {
|
||||
stats.get_me_writer_restored_fallback_total()
|
||||
} else {
|
||||
0
|
||||
}
|
||||
);
|
||||
|
||||
let unresolved_writer_losses = if me_allows_normal {
|
||||
stats
|
||||
.get_me_writer_removed_unexpected_total()
|
||||
.saturating_sub(
|
||||
stats
|
||||
.get_me_writer_restored_same_endpoint_total()
|
||||
.saturating_add(stats.get_me_writer_restored_fallback_total()),
|
||||
)
|
||||
} else {
|
||||
0
|
||||
};
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"# HELP telemt_me_writer_removed_unexpected_minus_restored_total Unexpected writer removals not yet compensated by restore"
|
||||
);
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"# TYPE telemt_me_writer_removed_unexpected_minus_restored_total gauge"
|
||||
);
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_me_writer_removed_unexpected_minus_restored_total {}",
|
||||
unresolved_writer_losses
|
||||
);
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_user_connections_total Per-user total connections");
|
||||
@@ -211,16 +653,63 @@ fn render_metrics(stats: &Stats) -> String {
|
||||
let _ = writeln!(out, "# TYPE telemt_user_msgs_from_client counter");
|
||||
let _ = writeln!(out, "# HELP telemt_user_msgs_to_client Per-user messages sent");
|
||||
let _ = writeln!(out, "# TYPE telemt_user_msgs_to_client counter");
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"# HELP telemt_telemetry_user_series_suppressed User-labeled metric series suppression flag"
|
||||
);
|
||||
let _ = writeln!(out, "# TYPE telemt_telemetry_user_series_suppressed gauge");
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_telemetry_user_series_suppressed {}",
|
||||
if user_enabled { 0 } else { 1 }
|
||||
);
|
||||
|
||||
for entry in stats.iter_user_stats() {
|
||||
let user = entry.key();
|
||||
let s = entry.value();
|
||||
let _ = writeln!(out, "telemt_user_connections_total{{user=\"{}\"}} {}", user, s.connects.load(std::sync::atomic::Ordering::Relaxed));
|
||||
let _ = writeln!(out, "telemt_user_connections_current{{user=\"{}\"}} {}", user, s.curr_connects.load(std::sync::atomic::Ordering::Relaxed));
|
||||
let _ = writeln!(out, "telemt_user_octets_from_client{{user=\"{}\"}} {}", user, s.octets_from_client.load(std::sync::atomic::Ordering::Relaxed));
|
||||
let _ = writeln!(out, "telemt_user_octets_to_client{{user=\"{}\"}} {}", user, s.octets_to_client.load(std::sync::atomic::Ordering::Relaxed));
|
||||
let _ = writeln!(out, "telemt_user_msgs_from_client{{user=\"{}\"}} {}", user, s.msgs_from_client.load(std::sync::atomic::Ordering::Relaxed));
|
||||
let _ = writeln!(out, "telemt_user_msgs_to_client{{user=\"{}\"}} {}", user, s.msgs_to_client.load(std::sync::atomic::Ordering::Relaxed));
|
||||
if user_enabled {
|
||||
for entry in stats.iter_user_stats() {
|
||||
let user = entry.key();
|
||||
let s = entry.value();
|
||||
let _ = writeln!(out, "telemt_user_connections_total{{user=\"{}\"}} {}", user, s.connects.load(std::sync::atomic::Ordering::Relaxed));
|
||||
let _ = writeln!(out, "telemt_user_connections_current{{user=\"{}\"}} {}", user, s.curr_connects.load(std::sync::atomic::Ordering::Relaxed));
|
||||
let _ = writeln!(out, "telemt_user_octets_from_client{{user=\"{}\"}} {}", user, s.octets_from_client.load(std::sync::atomic::Ordering::Relaxed));
|
||||
let _ = writeln!(out, "telemt_user_octets_to_client{{user=\"{}\"}} {}", user, s.octets_to_client.load(std::sync::atomic::Ordering::Relaxed));
|
||||
let _ = writeln!(out, "telemt_user_msgs_from_client{{user=\"{}\"}} {}", user, s.msgs_from_client.load(std::sync::atomic::Ordering::Relaxed));
|
||||
let _ = writeln!(out, "telemt_user_msgs_to_client{{user=\"{}\"}} {}", user, s.msgs_to_client.load(std::sync::atomic::Ordering::Relaxed));
|
||||
}
|
||||
|
||||
let ip_stats = ip_tracker.get_stats().await;
|
||||
let ip_counts: HashMap<String, usize> = ip_stats
|
||||
.into_iter()
|
||||
.map(|(user, count, _)| (user, count))
|
||||
.collect();
|
||||
|
||||
let mut unique_users = BTreeSet::new();
|
||||
unique_users.extend(config.access.user_max_unique_ips.keys().cloned());
|
||||
unique_users.extend(ip_counts.keys().cloned());
|
||||
|
||||
let _ = writeln!(out, "# HELP telemt_user_unique_ips_current Per-user current number of unique active IPs");
|
||||
let _ = writeln!(out, "# TYPE telemt_user_unique_ips_current gauge");
|
||||
let _ = writeln!(out, "# HELP telemt_user_unique_ips_limit Per-user configured unique IP limit (0 means unlimited)");
|
||||
let _ = writeln!(out, "# TYPE telemt_user_unique_ips_limit gauge");
|
||||
let _ = writeln!(out, "# HELP telemt_user_unique_ips_utilization Per-user unique IP usage ratio (0 for unlimited)");
|
||||
let _ = writeln!(out, "# TYPE telemt_user_unique_ips_utilization gauge");
|
||||
|
||||
for user in unique_users {
|
||||
let current = ip_counts.get(&user).copied().unwrap_or(0);
|
||||
let limit = config.access.user_max_unique_ips.get(&user).copied().unwrap_or(0);
|
||||
let utilization = if limit > 0 {
|
||||
current as f64 / limit as f64
|
||||
} else {
|
||||
0.0
|
||||
};
|
||||
let _ = writeln!(out, "telemt_user_unique_ips_current{{user=\"{}\"}} {}", user, current);
|
||||
let _ = writeln!(out, "telemt_user_unique_ips_limit{{user=\"{}\"}} {}", user, limit);
|
||||
let _ = writeln!(
|
||||
out,
|
||||
"telemt_user_unique_ips_utilization{{user=\"{}\"}} {:.6}",
|
||||
user,
|
||||
utilization
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
out
|
||||
@@ -229,10 +718,19 @@ fn render_metrics(stats: &Stats) -> String {
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::net::IpAddr;
|
||||
use http_body_util::BodyExt;
|
||||
|
||||
#[test]
|
||||
fn test_render_metrics_format() {
|
||||
#[tokio::test]
|
||||
async fn test_render_metrics_format() {
|
||||
let stats = Arc::new(Stats::new());
|
||||
let tracker = UserIpTracker::new();
|
||||
let mut config = ProxyConfig::default();
|
||||
config
|
||||
.access
|
||||
.user_max_unique_ips
|
||||
.insert("alice".to_string(), 4);
|
||||
|
||||
stats.increment_connects_all();
|
||||
stats.increment_connects_all();
|
||||
stats.increment_connects_bad();
|
||||
@@ -244,8 +742,12 @@ mod tests {
|
||||
stats.increment_user_msgs_from("alice");
|
||||
stats.increment_user_msgs_to("alice");
|
||||
stats.increment_user_msgs_to("alice");
|
||||
tracker
|
||||
.check_and_add("alice", "203.0.113.10".parse().unwrap())
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let output = render_metrics(&stats);
|
||||
let output = render_metrics(&stats, &config, &tracker).await;
|
||||
|
||||
assert!(output.contains("telemt_connections_total 2"));
|
||||
assert!(output.contains("telemt_connections_bad_total 1"));
|
||||
@@ -256,31 +758,48 @@ mod tests {
|
||||
assert!(output.contains("telemt_user_octets_to_client{user=\"alice\"} 2048"));
|
||||
assert!(output.contains("telemt_user_msgs_from_client{user=\"alice\"} 1"));
|
||||
assert!(output.contains("telemt_user_msgs_to_client{user=\"alice\"} 2"));
|
||||
assert!(output.contains("telemt_user_unique_ips_current{user=\"alice\"} 1"));
|
||||
assert!(output.contains("telemt_user_unique_ips_limit{user=\"alice\"} 4"));
|
||||
assert!(output.contains("telemt_user_unique_ips_utilization{user=\"alice\"} 0.250000"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_render_empty_stats() {
|
||||
#[tokio::test]
|
||||
async fn test_render_empty_stats() {
|
||||
let stats = Stats::new();
|
||||
let output = render_metrics(&stats);
|
||||
let tracker = UserIpTracker::new();
|
||||
let config = ProxyConfig::default();
|
||||
let output = render_metrics(&stats, &config, &tracker).await;
|
||||
assert!(output.contains("telemt_connections_total 0"));
|
||||
assert!(output.contains("telemt_connections_bad_total 0"));
|
||||
assert!(output.contains("telemt_handshake_timeouts_total 0"));
|
||||
assert!(!output.contains("user="));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_render_has_type_annotations() {
|
||||
#[tokio::test]
|
||||
async fn test_render_has_type_annotations() {
|
||||
let stats = Stats::new();
|
||||
let output = render_metrics(&stats);
|
||||
let tracker = UserIpTracker::new();
|
||||
let config = ProxyConfig::default();
|
||||
let output = render_metrics(&stats, &config, &tracker).await;
|
||||
assert!(output.contains("# TYPE telemt_uptime_seconds gauge"));
|
||||
assert!(output.contains("# TYPE telemt_connections_total counter"));
|
||||
assert!(output.contains("# TYPE telemt_connections_bad_total counter"));
|
||||
assert!(output.contains("# TYPE telemt_handshake_timeouts_total counter"));
|
||||
assert!(output.contains("# TYPE telemt_me_writer_removed_total counter"));
|
||||
assert!(output.contains(
|
||||
"# TYPE telemt_me_writer_removed_unexpected_minus_restored_total gauge"
|
||||
));
|
||||
assert!(output.contains("# TYPE telemt_user_unique_ips_current gauge"));
|
||||
assert!(output.contains("# TYPE telemt_user_unique_ips_limit gauge"));
|
||||
assert!(output.contains("# TYPE telemt_user_unique_ips_utilization gauge"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_endpoint_integration() {
|
||||
let stats = Arc::new(Stats::new());
|
||||
let beobachten = Arc::new(BeobachtenStore::new());
|
||||
let tracker = UserIpTracker::new();
|
||||
let mut config = ProxyConfig::default();
|
||||
stats.increment_connects_all();
|
||||
stats.increment_connects_all();
|
||||
stats.increment_connects_all();
|
||||
@@ -289,16 +808,38 @@ mod tests {
|
||||
.uri("/metrics")
|
||||
.body(())
|
||||
.unwrap();
|
||||
let resp = handle(req, &stats).unwrap();
|
||||
let resp = handle(req, &stats, &beobachten, &tracker, &config).await.unwrap();
|
||||
assert_eq!(resp.status(), StatusCode::OK);
|
||||
let body = resp.into_body().collect().await.unwrap().to_bytes();
|
||||
assert!(std::str::from_utf8(body.as_ref()).unwrap().contains("telemt_connections_total 3"));
|
||||
|
||||
config.general.beobachten = true;
|
||||
config.general.beobachten_minutes = 10;
|
||||
beobachten.record(
|
||||
"TLS-scanner",
|
||||
"203.0.113.10".parse::<IpAddr>().unwrap(),
|
||||
Duration::from_secs(600),
|
||||
);
|
||||
let req_beob = Request::builder()
|
||||
.uri("/beobachten")
|
||||
.body(())
|
||||
.unwrap();
|
||||
let resp_beob = handle(req_beob, &stats, &beobachten, &tracker, &config)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp_beob.status(), StatusCode::OK);
|
||||
let body_beob = resp_beob.into_body().collect().await.unwrap().to_bytes();
|
||||
let beob_text = std::str::from_utf8(body_beob.as_ref()).unwrap();
|
||||
assert!(beob_text.contains("[TLS-scanner]"));
|
||||
assert!(beob_text.contains("203.0.113.10-1"));
|
||||
|
||||
let req404 = Request::builder()
|
||||
.uri("/other")
|
||||
.body(())
|
||||
.unwrap();
|
||||
let resp404 = handle(req404, &stats).unwrap();
|
||||
let resp404 = handle(req404, &stats, &beobachten, &tracker, &config)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp404.status(), StatusCode::NOT_FOUND);
|
||||
}
|
||||
}
|
||||
|
||||
197
src/network/dns_overrides.rs
Normal file
197
src/network/dns_overrides.rs
Normal file
@@ -0,0 +1,197 @@
|
||||
//! Runtime DNS overrides for `host:port` targets.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::net::{IpAddr, Ipv6Addr, SocketAddr};
|
||||
use std::sync::{OnceLock, RwLock};
|
||||
|
||||
use crate::error::{ProxyError, Result};
|
||||
|
||||
type OverrideMap = HashMap<(String, u16), IpAddr>;
|
||||
|
||||
static DNS_OVERRIDES: OnceLock<RwLock<OverrideMap>> = OnceLock::new();
|
||||
|
||||
fn overrides_store() -> &'static RwLock<OverrideMap> {
|
||||
DNS_OVERRIDES.get_or_init(|| RwLock::new(HashMap::new()))
|
||||
}
|
||||
|
||||
fn parse_ip_spec(ip_spec: &str) -> Result<IpAddr> {
|
||||
if ip_spec.starts_with('[') && ip_spec.ends_with(']') {
|
||||
let inner = &ip_spec[1..ip_spec.len() - 1];
|
||||
let ipv6 = inner.parse::<Ipv6Addr>().map_err(|_| {
|
||||
ProxyError::Config(format!(
|
||||
"network.dns_overrides IPv6 override is invalid: '{ip_spec}'"
|
||||
))
|
||||
})?;
|
||||
return Ok(IpAddr::V6(ipv6));
|
||||
}
|
||||
|
||||
let ip = ip_spec.parse::<IpAddr>().map_err(|_| {
|
||||
ProxyError::Config(format!(
|
||||
"network.dns_overrides IP is invalid: '{ip_spec}'"
|
||||
))
|
||||
})?;
|
||||
if matches!(ip, IpAddr::V6(_)) {
|
||||
return Err(ProxyError::Config(format!(
|
||||
"network.dns_overrides IPv6 must be bracketed: '{ip_spec}'"
|
||||
)));
|
||||
}
|
||||
Ok(ip)
|
||||
}
|
||||
|
||||
fn parse_entry(entry: &str) -> Result<((String, u16), IpAddr)> {
|
||||
let trimmed = entry.trim();
|
||||
if trimmed.is_empty() {
|
||||
return Err(ProxyError::Config(
|
||||
"network.dns_overrides entry cannot be empty".to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
let first_sep = trimmed.find(':').ok_or_else(|| {
|
||||
ProxyError::Config(format!(
|
||||
"network.dns_overrides entry must use host:port:ip format: '{trimmed}'"
|
||||
))
|
||||
})?;
|
||||
let second_sep = trimmed[first_sep + 1..]
|
||||
.find(':')
|
||||
.map(|idx| first_sep + 1 + idx)
|
||||
.ok_or_else(|| {
|
||||
ProxyError::Config(format!(
|
||||
"network.dns_overrides entry must use host:port:ip format: '{trimmed}'"
|
||||
))
|
||||
})?;
|
||||
|
||||
let host = trimmed[..first_sep].trim();
|
||||
let port_str = trimmed[first_sep + 1..second_sep].trim();
|
||||
let ip_str = trimmed[second_sep + 1..].trim();
|
||||
|
||||
if host.is_empty() {
|
||||
return Err(ProxyError::Config(format!(
|
||||
"network.dns_overrides host cannot be empty: '{trimmed}'"
|
||||
)));
|
||||
}
|
||||
if host.contains(':') {
|
||||
return Err(ProxyError::Config(format!(
|
||||
"network.dns_overrides host must be a domain name without ':' in this format: '{trimmed}'"
|
||||
)));
|
||||
}
|
||||
|
||||
let port = port_str.parse::<u16>().map_err(|_| {
|
||||
ProxyError::Config(format!(
|
||||
"network.dns_overrides port is invalid: '{trimmed}'"
|
||||
))
|
||||
})?;
|
||||
let ip = parse_ip_spec(ip_str)?;
|
||||
|
||||
Ok(((host.to_ascii_lowercase(), port), ip))
|
||||
}
|
||||
|
||||
fn parse_entries(entries: &[String]) -> Result<OverrideMap> {
|
||||
let mut parsed = HashMap::new();
|
||||
for entry in entries {
|
||||
let (key, ip) = parse_entry(entry)?;
|
||||
parsed.insert(key, ip);
|
||||
}
|
||||
Ok(parsed)
|
||||
}
|
||||
|
||||
/// Validate `network.dns_overrides` entries without updating runtime state.
|
||||
pub fn validate_entries(entries: &[String]) -> Result<()> {
|
||||
let _ = parse_entries(entries)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Replace runtime DNS overrides with a new validated snapshot.
|
||||
pub fn install_entries(entries: &[String]) -> Result<()> {
|
||||
let parsed = parse_entries(entries)?;
|
||||
let mut guard = overrides_store()
|
||||
.write()
|
||||
.map_err(|_| ProxyError::Config("network.dns_overrides runtime lock is poisoned".to_string()))?;
|
||||
*guard = parsed;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Resolve a hostname override for `(host, port)` if present.
|
||||
pub fn resolve(host: &str, port: u16) -> Option<IpAddr> {
|
||||
let key = (host.to_ascii_lowercase(), port);
|
||||
overrides_store()
|
||||
.read()
|
||||
.ok()
|
||||
.and_then(|guard| guard.get(&key).copied())
|
||||
}
|
||||
|
||||
/// Resolve a hostname override and construct a socket address when present.
|
||||
pub fn resolve_socket_addr(host: &str, port: u16) -> Option<SocketAddr> {
|
||||
resolve(host, port).map(|ip| SocketAddr::new(ip, port))
|
||||
}
|
||||
|
||||
/// Parse a runtime endpoint in `host:port` format.
|
||||
///
|
||||
/// Supports:
|
||||
/// - `example.com:443`
|
||||
/// - `[2001:db8::1]:443`
|
||||
pub fn split_host_port(endpoint: &str) -> Option<(String, u16)> {
|
||||
if endpoint.starts_with('[') {
|
||||
let bracket_end = endpoint.find(']')?;
|
||||
if endpoint.as_bytes().get(bracket_end + 1) != Some(&b':') {
|
||||
return None;
|
||||
}
|
||||
let host = endpoint[1..bracket_end].trim();
|
||||
let port = endpoint[bracket_end + 2..].trim().parse::<u16>().ok()?;
|
||||
if host.is_empty() {
|
||||
return None;
|
||||
}
|
||||
return Some((host.to_ascii_lowercase(), port));
|
||||
}
|
||||
|
||||
let split_idx = endpoint.rfind(':')?;
|
||||
let host = endpoint[..split_idx].trim();
|
||||
let port = endpoint[split_idx + 1..].trim().parse::<u16>().ok()?;
|
||||
if host.is_empty() || host.contains(':') {
|
||||
return None;
|
||||
}
|
||||
|
||||
Some((host.to_ascii_lowercase(), port))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn validate_accepts_ipv4_and_bracketed_ipv6() {
|
||||
let entries = vec![
|
||||
"example.com:443:127.0.0.1".to_string(),
|
||||
"example.net:8443:[2001:db8::10]".to_string(),
|
||||
];
|
||||
assert!(validate_entries(&entries).is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn validate_rejects_unbracketed_ipv6() {
|
||||
let entries = vec!["example.net:443:2001:db8::10".to_string()];
|
||||
let err = validate_entries(&entries).unwrap_err().to_string();
|
||||
assert!(err.contains("must be bracketed"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn install_and_resolve_are_case_insensitive_for_host() {
|
||||
let entries = vec!["MyPetrovich.ru:8443:127.0.0.1".to_string()];
|
||||
install_entries(&entries).unwrap();
|
||||
|
||||
let resolved = resolve("mypetrovich.ru", 8443);
|
||||
assert_eq!(resolved, Some("127.0.0.1".parse().unwrap()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn split_host_port_parses_supported_shapes() {
|
||||
assert_eq!(
|
||||
split_host_port("example.com:443"),
|
||||
Some(("example.com".to_string(), 443))
|
||||
);
|
||||
assert_eq!(
|
||||
split_host_port("[2001:db8::1]:443"),
|
||||
Some(("2001:db8::1".to_string(), 443))
|
||||
);
|
||||
assert_eq!(split_host_port("2001:db8::1:443"), None);
|
||||
}
|
||||
}
|
||||
@@ -1,3 +1,4 @@
|
||||
pub mod dns_overrides;
|
||||
pub mod probe;
|
||||
pub mod stun;
|
||||
|
||||
|
||||
@@ -1,10 +1,16 @@
|
||||
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr, SocketAddr, UdpSocket};
|
||||
#![allow(dead_code)]
|
||||
|
||||
use tracing::{info, warn};
|
||||
use std::collections::HashMap;
|
||||
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr, SocketAddr, UdpSocket};
|
||||
use std::time::Duration;
|
||||
|
||||
use tokio::task::JoinSet;
|
||||
use tokio::time::timeout;
|
||||
use tracing::{debug, info, warn};
|
||||
|
||||
use crate::config::NetworkConfig;
|
||||
use crate::error::Result;
|
||||
use crate::network::stun::{stun_probe_dual, DualStunResult, IpFamily};
|
||||
use crate::network::stun::{stun_probe_dual, DualStunResult, IpFamily, StunProbeResult};
|
||||
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct NetworkProbe {
|
||||
@@ -47,7 +53,13 @@ impl NetworkDecision {
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn run_probe(config: &NetworkConfig, stun_addr: Option<String>, nat_probe: bool) -> Result<NetworkProbe> {
|
||||
const STUN_BATCH_TIMEOUT: Duration = Duration::from_secs(5);
|
||||
|
||||
pub async fn run_probe(
|
||||
config: &NetworkConfig,
|
||||
nat_probe: bool,
|
||||
stun_nat_probe_concurrency: usize,
|
||||
) -> Result<NetworkProbe> {
|
||||
let mut probe = NetworkProbe::default();
|
||||
|
||||
probe.detected_ipv4 = detect_local_ip_v4();
|
||||
@@ -56,21 +68,38 @@ pub async fn run_probe(config: &NetworkConfig, stun_addr: Option<String>, nat_pr
|
||||
probe.ipv4_is_bogon = probe.detected_ipv4.map(is_bogon_v4).unwrap_or(false);
|
||||
probe.ipv6_is_bogon = probe.detected_ipv6.map(is_bogon_v6).unwrap_or(false);
|
||||
|
||||
let stun_server = stun_addr.unwrap_or_else(|| "stun.l.google.com:19302".to_string());
|
||||
let stun_res = if nat_probe {
|
||||
match stun_probe_dual(&stun_server).await {
|
||||
Ok(res) => res,
|
||||
Err(e) => {
|
||||
warn!(error = %e, "STUN probe failed, continuing without reflection");
|
||||
DualStunResult::default()
|
||||
}
|
||||
let stun_res = if nat_probe && config.stun_use {
|
||||
let servers = collect_stun_servers(config);
|
||||
if servers.is_empty() {
|
||||
warn!("STUN probe is enabled but network.stun_servers is empty");
|
||||
DualStunResult::default()
|
||||
} else {
|
||||
probe_stun_servers_parallel(
|
||||
&servers,
|
||||
stun_nat_probe_concurrency.max(1),
|
||||
)
|
||||
.await
|
||||
}
|
||||
} else if nat_probe {
|
||||
info!("STUN probe is disabled by network.stun_use=false");
|
||||
DualStunResult::default()
|
||||
} else {
|
||||
DualStunResult::default()
|
||||
};
|
||||
probe.reflected_ipv4 = stun_res.v4.map(|r| r.reflected_addr);
|
||||
probe.reflected_ipv6 = stun_res.v6.map(|r| r.reflected_addr);
|
||||
|
||||
// If STUN is blocked but IPv4 is private, try HTTP public-IP fallback.
|
||||
if nat_probe
|
||||
&& probe.reflected_ipv4.is_none()
|
||||
&& probe.detected_ipv4.map(is_bogon_v4).unwrap_or(false)
|
||||
{
|
||||
if let Some(public_ip) = detect_public_ipv4_http(&config.http_ip_detect_urls).await {
|
||||
probe.reflected_ipv4 = Some(SocketAddr::new(IpAddr::V4(public_ip), 0));
|
||||
info!(public_ip = %public_ip, "STUN unavailable, using HTTP public IPv4 fallback");
|
||||
}
|
||||
}
|
||||
|
||||
probe.ipv4_nat_detected = match (probe.detected_ipv4, probe.reflected_ipv4) {
|
||||
(Some(det), Some(reflected)) => det != reflected.ip(),
|
||||
_ => false,
|
||||
@@ -92,24 +121,127 @@ pub async fn run_probe(config: &NetworkConfig, stun_addr: Option<String>, nat_pr
|
||||
Ok(probe)
|
||||
}
|
||||
|
||||
async fn detect_public_ipv4_http(urls: &[String]) -> Option<Ipv4Addr> {
|
||||
let client = reqwest::Client::builder()
|
||||
.timeout(Duration::from_secs(3))
|
||||
.build()
|
||||
.ok()?;
|
||||
|
||||
for url in urls {
|
||||
let response = match client.get(url).send().await {
|
||||
Ok(response) => response,
|
||||
Err(_) => continue,
|
||||
};
|
||||
|
||||
let body = match response.text().await {
|
||||
Ok(body) => body,
|
||||
Err(_) => continue,
|
||||
};
|
||||
|
||||
let Ok(ip) = body.trim().parse::<Ipv4Addr>() else {
|
||||
continue;
|
||||
};
|
||||
if !is_bogon_v4(ip) {
|
||||
return Some(ip);
|
||||
}
|
||||
}
|
||||
|
||||
None
|
||||
}
|
||||
|
||||
fn collect_stun_servers(config: &NetworkConfig) -> Vec<String> {
|
||||
let mut out = Vec::new();
|
||||
for s in &config.stun_servers {
|
||||
if !s.is_empty() && !out.contains(s) {
|
||||
out.push(s.clone());
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
async fn probe_stun_servers_parallel(
|
||||
servers: &[String],
|
||||
concurrency: usize,
|
||||
) -> DualStunResult {
|
||||
let mut join_set = JoinSet::new();
|
||||
let mut next_idx = 0usize;
|
||||
let mut best_v4_by_ip: HashMap<IpAddr, (usize, StunProbeResult)> = HashMap::new();
|
||||
let mut best_v6_by_ip: HashMap<IpAddr, (usize, StunProbeResult)> = HashMap::new();
|
||||
|
||||
while next_idx < servers.len() || !join_set.is_empty() {
|
||||
while next_idx < servers.len() && join_set.len() < concurrency {
|
||||
let stun_addr = servers[next_idx].clone();
|
||||
next_idx += 1;
|
||||
join_set.spawn(async move {
|
||||
let res = timeout(STUN_BATCH_TIMEOUT, stun_probe_dual(&stun_addr)).await;
|
||||
(stun_addr, res)
|
||||
});
|
||||
}
|
||||
|
||||
let Some(task) = join_set.join_next().await else {
|
||||
break;
|
||||
};
|
||||
|
||||
match task {
|
||||
Ok((stun_addr, Ok(Ok(result)))) => {
|
||||
if let Some(v4) = result.v4 {
|
||||
let entry = best_v4_by_ip.entry(v4.reflected_addr.ip()).or_insert((0, v4));
|
||||
entry.0 += 1;
|
||||
}
|
||||
if let Some(v6) = result.v6 {
|
||||
let entry = best_v6_by_ip.entry(v6.reflected_addr.ip()).or_insert((0, v6));
|
||||
entry.0 += 1;
|
||||
}
|
||||
if result.v4.is_some() || result.v6.is_some() {
|
||||
debug!(stun = %stun_addr, "STUN server responded within probe timeout");
|
||||
}
|
||||
}
|
||||
Ok((stun_addr, Ok(Err(e)))) => {
|
||||
debug!(error = %e, stun = %stun_addr, "STUN probe failed");
|
||||
}
|
||||
Ok((stun_addr, Err(_))) => {
|
||||
debug!(stun = %stun_addr, "STUN probe timeout");
|
||||
}
|
||||
Err(e) => {
|
||||
debug!(error = %e, "STUN probe task join failed");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let mut out = DualStunResult::default();
|
||||
if let Some((_, best)) = best_v4_by_ip
|
||||
.into_values()
|
||||
.max_by_key(|(count, _)| *count)
|
||||
{
|
||||
info!("STUN-Quorum reached, IP: {}", best.reflected_addr.ip());
|
||||
out.v4 = Some(best);
|
||||
}
|
||||
if let Some((_, best)) = best_v6_by_ip
|
||||
.into_values()
|
||||
.max_by_key(|(count, _)| *count)
|
||||
{
|
||||
info!("STUN-Quorum reached, IP: {}", best.reflected_addr.ip());
|
||||
out.v6 = Some(best);
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
pub fn decide_network_capabilities(config: &NetworkConfig, probe: &NetworkProbe) -> NetworkDecision {
|
||||
let mut decision = NetworkDecision::default();
|
||||
let ipv4_dc = config.ipv4 && probe.detected_ipv4.is_some();
|
||||
let ipv6_dc = config.ipv6.unwrap_or(probe.detected_ipv6.is_some()) && probe.detected_ipv6.is_some();
|
||||
|
||||
decision.ipv4_dc = config.ipv4 && probe.detected_ipv4.is_some();
|
||||
decision.ipv6_dc = config.ipv6.unwrap_or(probe.detected_ipv6.is_some()) && probe.detected_ipv6.is_some();
|
||||
|
||||
decision.ipv4_me = config.ipv4
|
||||
let ipv4_me = config.ipv4
|
||||
&& probe.detected_ipv4.is_some()
|
||||
&& (!probe.ipv4_is_bogon || probe.reflected_ipv4.is_some());
|
||||
|
||||
let ipv6_enabled = config.ipv6.unwrap_or(probe.detected_ipv6.is_some());
|
||||
decision.ipv6_me = ipv6_enabled
|
||||
let ipv6_me = ipv6_enabled
|
||||
&& probe.detected_ipv6.is_some()
|
||||
&& (!probe.ipv6_is_bogon || probe.reflected_ipv6.is_some());
|
||||
|
||||
decision.effective_prefer = match config.prefer {
|
||||
6 if decision.ipv6_me || decision.ipv6_dc => 6,
|
||||
4 if decision.ipv4_me || decision.ipv4_dc => 4,
|
||||
let effective_prefer = match config.prefer {
|
||||
6 if ipv6_me || ipv6_dc => 6,
|
||||
4 if ipv4_me || ipv4_dc => 4,
|
||||
6 => {
|
||||
warn!("prefer=6 requested but IPv6 unavailable; falling back to IPv4");
|
||||
4
|
||||
@@ -117,10 +249,17 @@ pub fn decide_network_capabilities(config: &NetworkConfig, probe: &NetworkProbe)
|
||||
_ => 4,
|
||||
};
|
||||
|
||||
let me_families = decision.ipv4_me as u8 + decision.ipv6_me as u8;
|
||||
decision.effective_multipath = config.multipath && me_families >= 2;
|
||||
let me_families = ipv4_me as u8 + ipv6_me as u8;
|
||||
let effective_multipath = config.multipath && me_families >= 2;
|
||||
|
||||
decision
|
||||
NetworkDecision {
|
||||
ipv4_dc,
|
||||
ipv6_dc,
|
||||
ipv4_me,
|
||||
ipv6_me,
|
||||
effective_prefer,
|
||||
effective_multipath,
|
||||
}
|
||||
}
|
||||
|
||||
fn detect_local_ip_v4() -> Option<Ipv4Addr> {
|
||||
|
||||
@@ -1,9 +1,13 @@
|
||||
#![allow(unreachable_code)]
|
||||
#![allow(dead_code)]
|
||||
|
||||
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr, SocketAddr};
|
||||
|
||||
use tokio::net::{lookup_host, UdpSocket};
|
||||
use tokio::time::{timeout, Duration, sleep};
|
||||
|
||||
use crate::error::{ProxyError, Result};
|
||||
use crate::network::dns_overrides::{resolve, split_host_port};
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
|
||||
pub enum IpFamily {
|
||||
@@ -37,16 +41,31 @@ pub async fn stun_probe_dual(stun_addr: &str) -> Result<DualStunResult> {
|
||||
}
|
||||
|
||||
pub async fn stun_probe_family(stun_addr: &str, family: IpFamily) -> Result<Option<StunProbeResult>> {
|
||||
stun_probe_family_with_bind(stun_addr, family, None).await
|
||||
}
|
||||
|
||||
pub async fn stun_probe_family_with_bind(
|
||||
stun_addr: &str,
|
||||
family: IpFamily,
|
||||
bind_ip: Option<IpAddr>,
|
||||
) -> Result<Option<StunProbeResult>> {
|
||||
use rand::RngCore;
|
||||
|
||||
let bind_addr = match family {
|
||||
IpFamily::V4 => "0.0.0.0:0",
|
||||
IpFamily::V6 => "[::]:0",
|
||||
let bind_addr = match (family, bind_ip) {
|
||||
(IpFamily::V4, Some(IpAddr::V4(ip))) => SocketAddr::new(IpAddr::V4(ip), 0),
|
||||
(IpFamily::V6, Some(IpAddr::V6(ip))) => SocketAddr::new(IpAddr::V6(ip), 0),
|
||||
(IpFamily::V4, Some(IpAddr::V6(_))) | (IpFamily::V6, Some(IpAddr::V4(_))) => {
|
||||
return Ok(None);
|
||||
}
|
||||
(IpFamily::V4, None) => SocketAddr::new(IpAddr::V4(Ipv4Addr::UNSPECIFIED), 0),
|
||||
(IpFamily::V6, None) => SocketAddr::new(IpAddr::V6(Ipv6Addr::UNSPECIFIED), 0),
|
||||
};
|
||||
|
||||
let socket = UdpSocket::bind(bind_addr)
|
||||
.await
|
||||
.map_err(|e| ProxyError::Proxy(format!("STUN bind failed: {e}")))?;
|
||||
let socket = match UdpSocket::bind(bind_addr).await {
|
||||
Ok(socket) => socket,
|
||||
Err(_) if bind_ip.is_some() => return Ok(None),
|
||||
Err(e) => return Err(ProxyError::Proxy(format!("STUN bind failed: {e}"))),
|
||||
};
|
||||
|
||||
let target_addr = resolve_stun_addr(stun_addr, family).await?;
|
||||
if let Some(addr) = target_addr {
|
||||
@@ -195,16 +214,21 @@ async fn resolve_stun_addr(stun_addr: &str, family: IpFamily) -> Result<Option<S
|
||||
});
|
||||
}
|
||||
|
||||
let addrs = lookup_host(stun_addr)
|
||||
if let Some((host, port)) = split_host_port(stun_addr)
|
||||
&& let Some(ip) = resolve(&host, port)
|
||||
{
|
||||
let addr = SocketAddr::new(ip, port);
|
||||
return Ok(match (addr.is_ipv4(), family) {
|
||||
(true, IpFamily::V4) | (false, IpFamily::V6) => Some(addr),
|
||||
_ => None,
|
||||
});
|
||||
}
|
||||
|
||||
let mut addrs = lookup_host(stun_addr)
|
||||
.await
|
||||
.map_err(|e| ProxyError::Proxy(format!("STUN resolve failed: {e}")))?;
|
||||
|
||||
let target = addrs
|
||||
.filter(|a| match (a.is_ipv4(), family) {
|
||||
(true, IpFamily::V4) => true,
|
||||
(false, IpFamily::V6) => true,
|
||||
_ => false,
|
||||
})
|
||||
.next();
|
||||
.find(|a| matches!((a.is_ipv4(), family), (true, IpFamily::V4) | (false, IpFamily::V6)));
|
||||
Ok(target)
|
||||
}
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
//! Protocol constants and datacenter addresses
|
||||
|
||||
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr};
|
||||
#![allow(dead_code)]
|
||||
|
||||
use std::net::{IpAddr, Ipv4Addr};
|
||||
|
||||
use crate::crypto::SecureRandom;
|
||||
use std::sync::LazyLock;
|
||||
@@ -158,7 +160,7 @@ pub const MAX_TLS_CHUNK_SIZE: usize = 16384 + 256;
|
||||
|
||||
/// Secure Intermediate payload is expected to be 4-byte aligned.
|
||||
pub fn is_valid_secure_payload_len(data_len: usize) -> bool {
|
||||
data_len % 4 == 0
|
||||
data_len.is_multiple_of(4)
|
||||
}
|
||||
|
||||
/// Compute Secure Intermediate payload length from wire length.
|
||||
@@ -177,7 +179,7 @@ pub fn secure_padding_len(data_len: usize, rng: &SecureRandom) -> usize {
|
||||
is_valid_secure_payload_len(data_len),
|
||||
"Secure payload must be 4-byte aligned, got {data_len}"
|
||||
);
|
||||
(rng.range(3) + 1) as usize
|
||||
rng.range(3) + 1
|
||||
}
|
||||
|
||||
// ============= Timeouts =============
|
||||
@@ -229,7 +231,6 @@ pub static RESERVED_NONCE_CONTINUES: &[[u8; 4]] = &[
|
||||
// ============= RPC Constants (for Middle Proxy) =============
|
||||
|
||||
/// RPC Proxy Request
|
||||
|
||||
/// RPC Flags (from Erlang mtp_rpc.erl)
|
||||
pub const RPC_FLAG_NOT_ENCRYPTED: u32 = 0x2;
|
||||
pub const RPC_FLAG_HAS_AD_TAG: u32 = 0x8;
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
//! MTProto frame types and metadata
|
||||
|
||||
#![allow(dead_code)]
|
||||
|
||||
use std::collections::HashMap;
|
||||
|
||||
/// Extra metadata associated with a frame
|
||||
@@ -83,7 +85,7 @@ impl FrameMode {
|
||||
pub fn validate_message_length(len: usize) -> bool {
|
||||
use super::constants::{MIN_MSG_LEN, MAX_MSG_LEN, PADDING_FILLER};
|
||||
|
||||
len >= MIN_MSG_LEN && len <= MAX_MSG_LEN && len % PADDING_FILLER.len() == 0
|
||||
(MIN_MSG_LEN..=MAX_MSG_LEN).contains(&len) && len.is_multiple_of(PADDING_FILLER.len())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
|
||||
@@ -5,7 +5,11 @@ pub mod frame;
|
||||
pub mod obfuscation;
|
||||
pub mod tls;
|
||||
|
||||
#[allow(unused_imports)]
|
||||
pub use constants::*;
|
||||
#[allow(unused_imports)]
|
||||
pub use frame::*;
|
||||
#[allow(unused_imports)]
|
||||
pub use obfuscation::*;
|
||||
#[allow(unused_imports)]
|
||||
pub use tls::*;
|
||||
@@ -1,8 +1,9 @@
|
||||
//! MTProto Obfuscation
|
||||
|
||||
#![allow(dead_code)]
|
||||
|
||||
use zeroize::Zeroize;
|
||||
use crate::crypto::{sha256, AesCtr};
|
||||
use crate::error::Result;
|
||||
use super::constants::*;
|
||||
|
||||
/// Obfuscation parameters from handshake
|
||||
|
||||
@@ -4,8 +4,11 @@
|
||||
//! for domain fronting. The handshake looks like valid TLS 1.3 but
|
||||
//! actually carries MTProto authentication data.
|
||||
|
||||
#![allow(dead_code)]
|
||||
|
||||
use crate::crypto::{sha256_hmac, SecureRandom};
|
||||
use crate::error::{ProxyError, Result};
|
||||
#[cfg(test)]
|
||||
use crate::error::ProxyError;
|
||||
use super::constants::*;
|
||||
use std::time::{SystemTime, UNIX_EPOCH};
|
||||
use num_bigint::BigUint;
|
||||
@@ -332,7 +335,7 @@ pub fn validate_tls_handshake(
|
||||
// This is a quirk in some clients that use uptime instead of real time
|
||||
let is_boot_time = timestamp < 60 * 60 * 24 * 1000; // < ~2.7 years in seconds
|
||||
|
||||
if !is_boot_time && (time_diff < TIME_SKEW_MIN || time_diff > TIME_SKEW_MAX) {
|
||||
if !is_boot_time && !(TIME_SKEW_MIN..=TIME_SKEW_MAX).contains(&time_diff) {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
@@ -390,7 +393,7 @@ pub fn build_server_hello(
|
||||
) -> Vec<u8> {
|
||||
const MIN_APP_DATA: usize = 64;
|
||||
const MAX_APP_DATA: usize = 16640; // RFC 8446 §5.2 upper bound
|
||||
let fake_cert_len = fake_cert_len.max(MIN_APP_DATA).min(MAX_APP_DATA);
|
||||
let fake_cert_len = fake_cert_len.clamp(MIN_APP_DATA, MAX_APP_DATA);
|
||||
let x25519_key = gen_fake_x25519_key(rng);
|
||||
|
||||
// Build ServerHello
|
||||
@@ -522,10 +525,10 @@ pub fn extract_sni_from_client_hello(handshake: &[u8]) -> Option<String> {
|
||||
if sn_pos + name_len > sn_end {
|
||||
break;
|
||||
}
|
||||
if name_type == 0 && name_len > 0 {
|
||||
if let Ok(host) = std::str::from_utf8(&handshake[sn_pos..sn_pos + name_len]) {
|
||||
return Some(host.to_string());
|
||||
}
|
||||
if name_type == 0 && name_len > 0
|
||||
&& let Ok(host) = std::str::from_utf8(&handshake[sn_pos..sn_pos + name_len])
|
||||
{
|
||||
return Some(host.to_string());
|
||||
}
|
||||
sn_pos += name_len;
|
||||
}
|
||||
@@ -568,7 +571,7 @@ pub fn extract_alpn_from_client_hello(handshake: &[u8]) -> Vec<Vec<u8>> {
|
||||
let list_len = u16::from_be_bytes([handshake[pos], handshake[pos+1]]) as usize;
|
||||
let mut lp = pos + 2;
|
||||
let list_end = (pos + 2).saturating_add(list_len).min(pos + elen);
|
||||
while lp + 1 <= list_end {
|
||||
while lp < list_end {
|
||||
let plen = handshake[lp] as usize;
|
||||
lp += 1;
|
||||
if lp + plen > list_end { break; }
|
||||
@@ -613,7 +616,7 @@ pub fn parse_tls_record_header(header: &[u8; 5]) -> Option<(u8, u16)> {
|
||||
///
|
||||
/// This is useful for testing that our ServerHello is well-formed.
|
||||
#[cfg(test)]
|
||||
fn validate_server_hello_structure(data: &[u8]) -> Result<()> {
|
||||
fn validate_server_hello_structure(data: &[u8]) -> Result<(), ProxyError> {
|
||||
if data.len() < 5 {
|
||||
return Err(ProxyError::InvalidTlsRecord {
|
||||
record_type: 0,
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
//! Client Handler
|
||||
|
||||
use std::future::Future;
|
||||
use std::net::SocketAddr;
|
||||
use std::net::{IpAddr, SocketAddr};
|
||||
use std::pin::Pin;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
@@ -27,6 +27,7 @@ use crate::error::{HandshakeResult, ProxyError, Result};
|
||||
use crate::ip_tracker::UserIpTracker;
|
||||
use crate::protocol::constants::*;
|
||||
use crate::protocol::tls;
|
||||
use crate::stats::beobachten::BeobachtenStore;
|
||||
use crate::stats::{ReplayChecker, Stats};
|
||||
use crate::stream::{BufferPool, CryptoReader, CryptoWriter};
|
||||
use crate::transport::middle_proxy::MePool;
|
||||
@@ -39,6 +40,36 @@ use crate::proxy::handshake::{HandshakeSuccess, handle_mtproto_handshake, handle
|
||||
use crate::proxy::masking::handle_bad_client;
|
||||
use crate::proxy::middle_relay::handle_via_middle_proxy;
|
||||
|
||||
fn beobachten_ttl(config: &ProxyConfig) -> Duration {
|
||||
Duration::from_secs(config.general.beobachten_minutes.saturating_mul(60))
|
||||
}
|
||||
|
||||
fn record_beobachten_class(
|
||||
beobachten: &BeobachtenStore,
|
||||
config: &ProxyConfig,
|
||||
peer_ip: IpAddr,
|
||||
class: &str,
|
||||
) {
|
||||
if !config.general.beobachten {
|
||||
return;
|
||||
}
|
||||
beobachten.record(class, peer_ip, beobachten_ttl(config));
|
||||
}
|
||||
|
||||
fn record_handshake_failure_class(
|
||||
beobachten: &BeobachtenStore,
|
||||
config: &ProxyConfig,
|
||||
peer_ip: IpAddr,
|
||||
error: &ProxyError,
|
||||
) {
|
||||
let class = if error.to_string().contains("expected 64 bytes, got 0") {
|
||||
"expected_64_got_0"
|
||||
} else {
|
||||
"other"
|
||||
};
|
||||
record_beobachten_class(beobachten, config, peer_ip, class);
|
||||
}
|
||||
|
||||
pub async fn handle_client_stream<S>(
|
||||
mut stream: S,
|
||||
peer: SocketAddr,
|
||||
@@ -51,6 +82,7 @@ pub async fn handle_client_stream<S>(
|
||||
me_pool: Option<Arc<MePool>>,
|
||||
tls_cache: Option<Arc<TlsFrontCache>>,
|
||||
ip_tracker: Arc<UserIpTracker>,
|
||||
beobachten: Arc<BeobachtenStore>,
|
||||
proxy_protocol_enabled: bool,
|
||||
) -> Result<()>
|
||||
where
|
||||
@@ -73,6 +105,7 @@ where
|
||||
Err(e) => {
|
||||
stats.increment_connects_bad();
|
||||
warn!(peer = %peer, error = %e, "Invalid PROXY protocol header");
|
||||
record_beobachten_class(&beobachten, &config, peer.ip(), "other");
|
||||
return Err(e);
|
||||
}
|
||||
}
|
||||
@@ -82,6 +115,9 @@ where
|
||||
|
||||
let handshake_timeout = Duration::from_secs(config.timeouts.client_handshake);
|
||||
let stats_for_timeout = stats.clone();
|
||||
let config_for_timeout = config.clone();
|
||||
let beobachten_for_timeout = beobachten.clone();
|
||||
let peer_for_timeout = real_peer.ip();
|
||||
|
||||
// For non-TCP streams, use a synthetic local address
|
||||
let local_addr: SocketAddr = format!("0.0.0.0:{}", config.server.port)
|
||||
@@ -103,7 +139,15 @@ where
|
||||
debug!(peer = %real_peer, tls_len = tls_len, "TLS handshake too short");
|
||||
stats.increment_connects_bad();
|
||||
let (reader, writer) = tokio::io::split(stream);
|
||||
handle_bad_client(reader, writer, &first_bytes, &config).await;
|
||||
handle_bad_client(
|
||||
reader,
|
||||
writer,
|
||||
&first_bytes,
|
||||
real_peer,
|
||||
&config,
|
||||
&beobachten,
|
||||
)
|
||||
.await;
|
||||
return Ok(HandshakeOutcome::Handled);
|
||||
}
|
||||
|
||||
@@ -120,7 +164,15 @@ where
|
||||
HandshakeResult::Success(result) => result,
|
||||
HandshakeResult::BadClient { reader, writer } => {
|
||||
stats.increment_connects_bad();
|
||||
handle_bad_client(reader, writer, &handshake, &config).await;
|
||||
handle_bad_client(
|
||||
reader,
|
||||
writer,
|
||||
&handshake,
|
||||
real_peer,
|
||||
&config,
|
||||
&beobachten,
|
||||
)
|
||||
.await;
|
||||
return Ok(HandshakeOutcome::Handled);
|
||||
}
|
||||
HandshakeResult::Error(e) => return Err(e),
|
||||
@@ -156,7 +208,15 @@ where
|
||||
debug!(peer = %real_peer, "Non-TLS modes disabled");
|
||||
stats.increment_connects_bad();
|
||||
let (reader, writer) = tokio::io::split(stream);
|
||||
handle_bad_client(reader, writer, &first_bytes, &config).await;
|
||||
handle_bad_client(
|
||||
reader,
|
||||
writer,
|
||||
&first_bytes,
|
||||
real_peer,
|
||||
&config,
|
||||
&beobachten,
|
||||
)
|
||||
.await;
|
||||
return Ok(HandshakeOutcome::Handled);
|
||||
}
|
||||
|
||||
@@ -173,7 +233,15 @@ where
|
||||
HandshakeResult::Success(result) => result,
|
||||
HandshakeResult::BadClient { reader, writer } => {
|
||||
stats.increment_connects_bad();
|
||||
handle_bad_client(reader, writer, &handshake, &config).await;
|
||||
handle_bad_client(
|
||||
reader,
|
||||
writer,
|
||||
&handshake,
|
||||
real_peer,
|
||||
&config,
|
||||
&beobachten,
|
||||
)
|
||||
.await;
|
||||
return Ok(HandshakeOutcome::Handled);
|
||||
}
|
||||
HandshakeResult::Error(e) => return Err(e),
|
||||
@@ -200,11 +268,23 @@ where
|
||||
Ok(Ok(outcome)) => outcome,
|
||||
Ok(Err(e)) => {
|
||||
debug!(peer = %peer, error = %e, "Handshake failed");
|
||||
record_handshake_failure_class(
|
||||
&beobachten_for_timeout,
|
||||
&config_for_timeout,
|
||||
peer_for_timeout,
|
||||
&e,
|
||||
);
|
||||
return Err(e);
|
||||
}
|
||||
Err(_) => {
|
||||
stats_for_timeout.increment_handshake_timeouts();
|
||||
debug!(peer = %peer, "Handshake timeout");
|
||||
record_beobachten_class(
|
||||
&beobachten_for_timeout,
|
||||
&config_for_timeout,
|
||||
peer_for_timeout,
|
||||
"other",
|
||||
);
|
||||
return Err(ProxyError::TgHandshakeTimeout);
|
||||
}
|
||||
};
|
||||
@@ -230,6 +310,7 @@ pub struct RunningClientHandler {
|
||||
me_pool: Option<Arc<MePool>>,
|
||||
tls_cache: Option<Arc<TlsFrontCache>>,
|
||||
ip_tracker: Arc<UserIpTracker>,
|
||||
beobachten: Arc<BeobachtenStore>,
|
||||
proxy_protocol_enabled: bool,
|
||||
}
|
||||
|
||||
@@ -246,6 +327,7 @@ impl ClientHandler {
|
||||
me_pool: Option<Arc<MePool>>,
|
||||
tls_cache: Option<Arc<TlsFrontCache>>,
|
||||
ip_tracker: Arc<UserIpTracker>,
|
||||
beobachten: Arc<BeobachtenStore>,
|
||||
proxy_protocol_enabled: bool,
|
||||
) -> RunningClientHandler {
|
||||
RunningClientHandler {
|
||||
@@ -260,6 +342,7 @@ impl ClientHandler {
|
||||
me_pool,
|
||||
tls_cache,
|
||||
ip_tracker,
|
||||
beobachten,
|
||||
proxy_protocol_enabled,
|
||||
}
|
||||
}
|
||||
@@ -271,7 +354,7 @@ impl RunningClientHandler {
|
||||
|
||||
self.peer = normalize_ip(self.peer);
|
||||
let peer = self.peer;
|
||||
let ip_tracker = self.ip_tracker.clone();
|
||||
let _ip_tracker = self.ip_tracker.clone();
|
||||
debug!(peer = %peer, "New connection");
|
||||
|
||||
if let Err(e) = configure_client_socket(
|
||||
@@ -284,17 +367,32 @@ impl RunningClientHandler {
|
||||
|
||||
let handshake_timeout = Duration::from_secs(self.config.timeouts.client_handshake);
|
||||
let stats = self.stats.clone();
|
||||
let config_for_timeout = self.config.clone();
|
||||
let beobachten_for_timeout = self.beobachten.clone();
|
||||
let peer_for_timeout = peer.ip();
|
||||
|
||||
// Phase 1: handshake (with timeout)
|
||||
let outcome = match timeout(handshake_timeout, self.do_handshake()).await {
|
||||
Ok(Ok(outcome)) => outcome,
|
||||
Ok(Err(e)) => {
|
||||
debug!(peer = %peer, error = %e, "Handshake failed");
|
||||
record_handshake_failure_class(
|
||||
&beobachten_for_timeout,
|
||||
&config_for_timeout,
|
||||
peer_for_timeout,
|
||||
&e,
|
||||
);
|
||||
return Err(e);
|
||||
}
|
||||
Err(_) => {
|
||||
stats.increment_handshake_timeouts();
|
||||
debug!(peer = %peer, "Handshake timeout");
|
||||
record_beobachten_class(
|
||||
&beobachten_for_timeout,
|
||||
&config_for_timeout,
|
||||
peer_for_timeout,
|
||||
"other",
|
||||
);
|
||||
return Err(ProxyError::TgHandshakeTimeout);
|
||||
}
|
||||
};
|
||||
@@ -321,6 +419,12 @@ impl RunningClientHandler {
|
||||
Err(e) => {
|
||||
self.stats.increment_connects_bad();
|
||||
warn!(peer = %self.peer, error = %e, "Invalid PROXY protocol header");
|
||||
record_beobachten_class(
|
||||
&self.beobachten,
|
||||
&self.config,
|
||||
self.peer.ip(),
|
||||
"other",
|
||||
);
|
||||
return Err(e);
|
||||
}
|
||||
}
|
||||
@@ -331,7 +435,7 @@ impl RunningClientHandler {
|
||||
|
||||
let is_tls = tls::is_tls_handshake(&first_bytes[..3]);
|
||||
let peer = self.peer;
|
||||
let ip_tracker = self.ip_tracker.clone();
|
||||
let _ip_tracker = self.ip_tracker.clone();
|
||||
|
||||
debug!(peer = %peer, is_tls = is_tls, "Handshake type detected");
|
||||
|
||||
@@ -344,7 +448,7 @@ impl RunningClientHandler {
|
||||
|
||||
async fn handle_tls_client(mut self, first_bytes: [u8; 5]) -> Result<HandshakeOutcome> {
|
||||
let peer = self.peer;
|
||||
let ip_tracker = self.ip_tracker.clone();
|
||||
let _ip_tracker = self.ip_tracker.clone();
|
||||
|
||||
let tls_len = u16::from_be_bytes([first_bytes[3], first_bytes[4]]) as usize;
|
||||
|
||||
@@ -354,7 +458,15 @@ impl RunningClientHandler {
|
||||
debug!(peer = %peer, tls_len = tls_len, "TLS handshake too short");
|
||||
self.stats.increment_connects_bad();
|
||||
let (reader, writer) = self.stream.into_split();
|
||||
handle_bad_client(reader, writer, &first_bytes, &self.config).await;
|
||||
handle_bad_client(
|
||||
reader,
|
||||
writer,
|
||||
&first_bytes,
|
||||
peer,
|
||||
&self.config,
|
||||
&self.beobachten,
|
||||
)
|
||||
.await;
|
||||
return Ok(HandshakeOutcome::Handled);
|
||||
}
|
||||
|
||||
@@ -385,7 +497,15 @@ impl RunningClientHandler {
|
||||
HandshakeResult::Success(result) => result,
|
||||
HandshakeResult::BadClient { reader, writer } => {
|
||||
stats.increment_connects_bad();
|
||||
handle_bad_client(reader, writer, &handshake, &config).await;
|
||||
handle_bad_client(
|
||||
reader,
|
||||
writer,
|
||||
&handshake,
|
||||
peer,
|
||||
&config,
|
||||
&self.beobachten,
|
||||
)
|
||||
.await;
|
||||
return Ok(HandshakeOutcome::Handled);
|
||||
}
|
||||
HandshakeResult::Error(e) => return Err(e),
|
||||
@@ -440,13 +560,21 @@ impl RunningClientHandler {
|
||||
|
||||
async fn handle_direct_client(mut self, first_bytes: [u8; 5]) -> Result<HandshakeOutcome> {
|
||||
let peer = self.peer;
|
||||
let ip_tracker = self.ip_tracker.clone();
|
||||
let _ip_tracker = self.ip_tracker.clone();
|
||||
|
||||
if !self.config.general.modes.classic && !self.config.general.modes.secure {
|
||||
debug!(peer = %peer, "Non-TLS modes disabled");
|
||||
self.stats.increment_connects_bad();
|
||||
let (reader, writer) = self.stream.into_split();
|
||||
handle_bad_client(reader, writer, &first_bytes, &self.config).await;
|
||||
handle_bad_client(
|
||||
reader,
|
||||
writer,
|
||||
&first_bytes,
|
||||
peer,
|
||||
&self.config,
|
||||
&self.beobachten,
|
||||
)
|
||||
.await;
|
||||
return Ok(HandshakeOutcome::Handled);
|
||||
}
|
||||
|
||||
@@ -476,7 +604,15 @@ impl RunningClientHandler {
|
||||
HandshakeResult::Success(result) => result,
|
||||
HandshakeResult::BadClient { reader, writer } => {
|
||||
stats.increment_connects_bad();
|
||||
handle_bad_client(reader, writer, &handshake, &config).await;
|
||||
handle_bad_client(
|
||||
reader,
|
||||
writer,
|
||||
&handshake,
|
||||
peer,
|
||||
&config,
|
||||
&self.beobachten,
|
||||
)
|
||||
.await;
|
||||
return Ok(HandshakeOutcome::Handled);
|
||||
}
|
||||
HandshakeResult::Error(e) => return Err(e),
|
||||
@@ -594,18 +730,18 @@ impl RunningClientHandler {
|
||||
peer_addr: SocketAddr,
|
||||
ip_tracker: &UserIpTracker,
|
||||
) -> Result<()> {
|
||||
if let Some(expiration) = config.access.user_expirations.get(user) {
|
||||
if chrono::Utc::now() > *expiration {
|
||||
return Err(ProxyError::UserExpired {
|
||||
user: user.to_string(),
|
||||
});
|
||||
}
|
||||
if let Some(expiration) = config.access.user_expirations.get(user)
|
||||
&& chrono::Utc::now() > *expiration
|
||||
{
|
||||
return Err(ProxyError::UserExpired {
|
||||
user: user.to_string(),
|
||||
});
|
||||
}
|
||||
|
||||
// IP limit check
|
||||
if let Err(reason) = ip_tracker.check_and_add(user, peer_addr.ip()).await {
|
||||
warn!(
|
||||
user = %user,
|
||||
user = %user,
|
||||
ip = %peer_addr.ip(),
|
||||
reason = %reason,
|
||||
"IP limit exceeded"
|
||||
@@ -615,20 +751,20 @@ impl RunningClientHandler {
|
||||
});
|
||||
}
|
||||
|
||||
if let Some(limit) = config.access.user_max_tcp_conns.get(user) {
|
||||
if stats.get_user_curr_connects(user) >= *limit as u64 {
|
||||
return Err(ProxyError::ConnectionLimitExceeded {
|
||||
user: user.to_string(),
|
||||
});
|
||||
}
|
||||
if let Some(limit) = config.access.user_max_tcp_conns.get(user)
|
||||
&& stats.get_user_curr_connects(user) >= *limit as u64
|
||||
{
|
||||
return Err(ProxyError::ConnectionLimitExceeded {
|
||||
user: user.to_string(),
|
||||
});
|
||||
}
|
||||
|
||||
if let Some(quota) = config.access.user_data_quota.get(user) {
|
||||
if stats.get_user_total_octets(user) >= *quota {
|
||||
return Err(ProxyError::DataQuotaExceeded {
|
||||
user: user.to_string(),
|
||||
});
|
||||
}
|
||||
if let Some(quota) = config.access.user_data_quota.get(user)
|
||||
&& stats.get_user_total_octets(user) >= *quota
|
||||
{
|
||||
return Err(ProxyError::DataQuotaExceeded {
|
||||
user: user.to_string(),
|
||||
});
|
||||
}
|
||||
|
||||
Ok(())
|
||||
|
||||
@@ -118,10 +118,10 @@ fn get_dc_addr_static(dc_idx: i16, config: &ProxyConfig) -> Result<SocketAddr> {
|
||||
// Unknown DC requested by client without override: log and fall back.
|
||||
if !config.dc_overrides.contains_key(&dc_key) {
|
||||
warn!(dc_idx = dc_idx, "Requested non-standard DC with no override; falling back to default cluster");
|
||||
if let Some(path) = &config.general.unknown_dc_log_path {
|
||||
if let Ok(mut file) = OpenOptions::new().create(true).append(true).open(path) {
|
||||
let _ = writeln!(file, "dc_idx={dc_idx}");
|
||||
}
|
||||
if let Some(path) = &config.general.unknown_dc_log_path
|
||||
&& let Ok(mut file) = OpenOptions::new().create(true).append(true).open(path)
|
||||
{
|
||||
let _ = writeln!(file, "dc_idx={dc_idx}");
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
//! MTProto Handshake
|
||||
|
||||
#![allow(dead_code)]
|
||||
|
||||
use std::net::SocketAddr;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
//! Masking - forward unrecognized traffic to mask host
|
||||
|
||||
use std::str;
|
||||
use std::net::SocketAddr;
|
||||
use std::time::Duration;
|
||||
use tokio::net::TcpStream;
|
||||
#[cfg(unix)]
|
||||
@@ -9,6 +10,9 @@ use tokio::io::{AsyncRead, AsyncWrite, AsyncReadExt, AsyncWriteExt};
|
||||
use tokio::time::timeout;
|
||||
use tracing::debug;
|
||||
use crate::config::ProxyConfig;
|
||||
use crate::network::dns_overrides::resolve_socket_addr;
|
||||
use crate::stats::beobachten::BeobachtenStore;
|
||||
use crate::transport::proxy_protocol::{ProxyProtocolV1Builder, ProxyProtocolV2Builder};
|
||||
|
||||
const MASK_TIMEOUT: Duration = Duration::from_secs(5);
|
||||
/// Maximum duration for the entire masking relay.
|
||||
@@ -19,12 +23,12 @@ const MASK_BUFFER_SIZE: usize = 8192;
|
||||
/// Detect client type based on initial data
|
||||
fn detect_client_type(data: &[u8]) -> &'static str {
|
||||
// Check for HTTP request
|
||||
if data.len() > 4 {
|
||||
if data.starts_with(b"GET ") || data.starts_with(b"POST") ||
|
||||
if data.len() > 4
|
||||
&& (data.starts_with(b"GET ") || data.starts_with(b"POST") ||
|
||||
data.starts_with(b"HEAD") || data.starts_with(b"PUT ") ||
|
||||
data.starts_with(b"DELETE") || data.starts_with(b"OPTIONS") {
|
||||
return "HTTP";
|
||||
}
|
||||
data.starts_with(b"DELETE") || data.starts_with(b"OPTIONS"))
|
||||
{
|
||||
return "HTTP";
|
||||
}
|
||||
|
||||
// Check for TLS ClientHello (0x16 = handshake, 0x03 0x01-0x03 = TLS version)
|
||||
@@ -50,20 +54,26 @@ pub async fn handle_bad_client<R, W>(
|
||||
reader: R,
|
||||
writer: W,
|
||||
initial_data: &[u8],
|
||||
peer: SocketAddr,
|
||||
config: &ProxyConfig,
|
||||
beobachten: &BeobachtenStore,
|
||||
)
|
||||
where
|
||||
R: AsyncRead + Unpin + Send + 'static,
|
||||
W: AsyncWrite + Unpin + Send + 'static,
|
||||
{
|
||||
let client_type = detect_client_type(initial_data);
|
||||
if config.general.beobachten {
|
||||
let ttl = Duration::from_secs(config.general.beobachten_minutes.saturating_mul(60));
|
||||
beobachten.record(client_type, peer.ip(), ttl);
|
||||
}
|
||||
|
||||
if !config.censorship.mask {
|
||||
// Masking disabled, just consume data
|
||||
consume_client_data(reader).await;
|
||||
return;
|
||||
}
|
||||
|
||||
let client_type = detect_client_type(initial_data);
|
||||
|
||||
// Connect via Unix socket or TCP
|
||||
#[cfg(unix)]
|
||||
if let Some(ref sock_path) = config.censorship.mask_unix_sock {
|
||||
@@ -106,12 +116,44 @@ where
|
||||
"Forwarding bad client to mask host"
|
||||
);
|
||||
|
||||
// Connect to mask host
|
||||
let mask_addr = format!("{}:{}", mask_host, mask_port);
|
||||
// Apply runtime DNS override for mask target when configured.
|
||||
let mask_addr = resolve_socket_addr(mask_host, mask_port)
|
||||
.map(|addr| addr.to_string())
|
||||
.unwrap_or_else(|| format!("{}:{}", mask_host, mask_port));
|
||||
let connect_result = timeout(MASK_TIMEOUT, TcpStream::connect(&mask_addr)).await;
|
||||
match connect_result {
|
||||
Ok(Ok(stream)) => {
|
||||
let (mask_read, mask_write) = stream.into_split();
|
||||
let proxy_header: Option<Vec<u8>> = match config.censorship.mask_proxy_protocol {
|
||||
0 => None,
|
||||
version => {
|
||||
let header = if let Ok(local_addr) = stream.local_addr() {
|
||||
match version {
|
||||
2 => ProxyProtocolV2Builder::new().with_addrs(peer, local_addr).build(),
|
||||
_ => match (peer, local_addr) {
|
||||
(SocketAddr::V4(src), SocketAddr::V4(dst)) =>
|
||||
ProxyProtocolV1Builder::new().tcp4(src.into(), dst.into()).build(),
|
||||
(SocketAddr::V6(src), SocketAddr::V6(dst)) =>
|
||||
ProxyProtocolV1Builder::new().tcp6(src.into(), dst.into()).build(),
|
||||
_ =>
|
||||
ProxyProtocolV1Builder::new().build(),
|
||||
},
|
||||
}
|
||||
} else {
|
||||
match version {
|
||||
2 => ProxyProtocolV2Builder::new().build(),
|
||||
_ => ProxyProtocolV1Builder::new().build(),
|
||||
}
|
||||
};
|
||||
Some(header)
|
||||
}
|
||||
};
|
||||
|
||||
let (mask_read, mut mask_write) = stream.into_split();
|
||||
if let Some(header) = proxy_header {
|
||||
if mask_write.write_all(&header).await.is_err() {
|
||||
return;
|
||||
}
|
||||
}
|
||||
if timeout(MASK_RELAY_TIMEOUT, relay_to_mask(reader, writer, mask_read, mask_write, initial_data)).await.is_err() {
|
||||
debug!("Mask relay timed out");
|
||||
}
|
||||
|
||||
@@ -393,13 +393,13 @@ where
|
||||
.unwrap_or_else(|e| Err(ProxyError::Proxy(format!("ME writer join error: {e}"))));
|
||||
|
||||
// When client closes, but ME channel stopped as unregistered - it isnt error
|
||||
if client_closed {
|
||||
if matches!(
|
||||
if client_closed
|
||||
&& matches!(
|
||||
writer_result,
|
||||
Err(ProxyError::Proxy(ref msg)) if msg == "ME connection lost"
|
||||
) {
|
||||
writer_result = Ok(());
|
||||
}
|
||||
)
|
||||
{
|
||||
writer_result = Ok(());
|
||||
}
|
||||
|
||||
let result = match (main_result, c2me_result, writer_result) {
|
||||
@@ -549,7 +549,7 @@ where
|
||||
|
||||
match proto_tag {
|
||||
ProtoTag::Abridged => {
|
||||
if data.len() % 4 != 0 {
|
||||
if !data.len().is_multiple_of(4) {
|
||||
return Err(ProxyError::Proxy(format!(
|
||||
"Abridged payload must be 4-byte aligned, got {}",
|
||||
data.len()
|
||||
@@ -567,7 +567,7 @@ where
|
||||
frame_buf.push(first);
|
||||
frame_buf.extend_from_slice(data);
|
||||
client_writer
|
||||
.write_all(&frame_buf)
|
||||
.write_all(frame_buf)
|
||||
.await
|
||||
.map_err(ProxyError::Io)?;
|
||||
} else if len_words < (1 << 24) {
|
||||
@@ -581,7 +581,7 @@ where
|
||||
frame_buf.extend_from_slice(&[first, lw[0], lw[1], lw[2]]);
|
||||
frame_buf.extend_from_slice(data);
|
||||
client_writer
|
||||
.write_all(&frame_buf)
|
||||
.write_all(frame_buf)
|
||||
.await
|
||||
.map_err(ProxyError::Io)?;
|
||||
} else {
|
||||
@@ -618,7 +618,7 @@ where
|
||||
rng.fill(&mut frame_buf[start..]);
|
||||
}
|
||||
client_writer
|
||||
.write_all(&frame_buf)
|
||||
.write_all(frame_buf)
|
||||
.await
|
||||
.map_err(ProxyError::Io)?;
|
||||
}
|
||||
|
||||
@@ -8,6 +8,9 @@ pub mod middle_relay;
|
||||
pub mod relay;
|
||||
|
||||
pub use client::ClientHandler;
|
||||
#[allow(unused_imports)]
|
||||
pub use handshake::*;
|
||||
#[allow(unused_imports)]
|
||||
pub use masking::*;
|
||||
#[allow(unused_imports)]
|
||||
pub use relay::*;
|
||||
|
||||
117
src/stats/beobachten.rs
Normal file
117
src/stats/beobachten.rs
Normal file
@@ -0,0 +1,117 @@
|
||||
//! Per-IP forensic buckets for scanner and handshake failure observation.
|
||||
|
||||
use std::collections::{BTreeMap, HashMap};
|
||||
use std::net::IpAddr;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use parking_lot::Mutex;
|
||||
|
||||
const CLEANUP_INTERVAL: Duration = Duration::from_secs(30);
|
||||
|
||||
#[derive(Default)]
|
||||
struct BeobachtenInner {
|
||||
entries: HashMap<(String, IpAddr), BeobachtenEntry>,
|
||||
last_cleanup: Option<Instant>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy)]
|
||||
struct BeobachtenEntry {
|
||||
tries: u64,
|
||||
last_seen: Instant,
|
||||
}
|
||||
|
||||
/// In-memory, TTL-scoped per-IP counters keyed by source class.
|
||||
pub struct BeobachtenStore {
|
||||
inner: Mutex<BeobachtenInner>,
|
||||
}
|
||||
|
||||
impl Default for BeobachtenStore {
|
||||
fn default() -> Self {
|
||||
Self::new()
|
||||
}
|
||||
}
|
||||
|
||||
impl BeobachtenStore {
|
||||
pub fn new() -> Self {
|
||||
Self {
|
||||
inner: Mutex::new(BeobachtenInner::default()),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn record(&self, class: &str, ip: IpAddr, ttl: Duration) {
|
||||
if class.is_empty() || ttl.is_zero() {
|
||||
return;
|
||||
}
|
||||
|
||||
let now = Instant::now();
|
||||
let mut guard = self.inner.lock();
|
||||
Self::cleanup_if_needed(&mut guard, now, ttl);
|
||||
|
||||
let key = (class.to_string(), ip);
|
||||
let entry = guard.entries.entry(key).or_insert(BeobachtenEntry {
|
||||
tries: 0,
|
||||
last_seen: now,
|
||||
});
|
||||
entry.tries = entry.tries.saturating_add(1);
|
||||
entry.last_seen = now;
|
||||
}
|
||||
|
||||
pub fn snapshot_text(&self, ttl: Duration) -> String {
|
||||
if ttl.is_zero() {
|
||||
return "beobachten disabled\n".to_string();
|
||||
}
|
||||
|
||||
let now = Instant::now();
|
||||
let mut guard = self.inner.lock();
|
||||
Self::cleanup(&mut guard, now, ttl);
|
||||
guard.last_cleanup = Some(now);
|
||||
|
||||
let mut grouped = BTreeMap::<String, Vec<(IpAddr, u64)>>::new();
|
||||
for ((class, ip), entry) in &guard.entries {
|
||||
grouped
|
||||
.entry(class.clone())
|
||||
.or_default()
|
||||
.push((*ip, entry.tries));
|
||||
}
|
||||
|
||||
if grouped.is_empty() {
|
||||
return "empty\n".to_string();
|
||||
}
|
||||
|
||||
let mut out = String::with_capacity(grouped.len() * 64);
|
||||
for (class, entries) in &mut grouped {
|
||||
out.push('[');
|
||||
out.push_str(class);
|
||||
out.push_str("]\n");
|
||||
|
||||
entries.sort_by(|(ip_a, tries_a), (ip_b, tries_b)| {
|
||||
tries_b
|
||||
.cmp(tries_a)
|
||||
.then_with(|| ip_a.to_string().cmp(&ip_b.to_string()))
|
||||
});
|
||||
|
||||
for (ip, tries) in entries {
|
||||
out.push_str(&format!("{ip}-{tries}\n"));
|
||||
}
|
||||
}
|
||||
|
||||
out
|
||||
}
|
||||
|
||||
fn cleanup_if_needed(inner: &mut BeobachtenInner, now: Instant, ttl: Duration) {
|
||||
let should_cleanup = match inner.last_cleanup {
|
||||
Some(last) => now.saturating_duration_since(last) >= CLEANUP_INTERVAL,
|
||||
None => true,
|
||||
};
|
||||
if should_cleanup {
|
||||
Self::cleanup(inner, now, ttl);
|
||||
inner.last_cleanup = Some(now);
|
||||
}
|
||||
}
|
||||
|
||||
fn cleanup(inner: &mut BeobachtenInner, now: Instant, ttl: Duration) {
|
||||
inner.entries.retain(|_, entry| {
|
||||
now.saturating_duration_since(entry.last_seen) <= ttl
|
||||
});
|
||||
}
|
||||
}
|
||||
341
src/stats/mod.rs
341
src/stats/mod.rs
@@ -1,7 +1,11 @@
|
||||
//! Statistics and replay protection
|
||||
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::sync::Arc;
|
||||
#![allow(dead_code)]
|
||||
|
||||
pub mod beobachten;
|
||||
pub mod telemetry;
|
||||
|
||||
use std::sync::atomic::{AtomicBool, AtomicU8, AtomicU64, Ordering};
|
||||
use std::time::{Instant, Duration};
|
||||
use dashmap::DashMap;
|
||||
use parking_lot::Mutex;
|
||||
@@ -12,6 +16,9 @@ use std::collections::hash_map::DefaultHasher;
|
||||
use std::collections::VecDeque;
|
||||
use tracing::debug;
|
||||
|
||||
use crate::config::MeTelemetryLevel;
|
||||
use self::telemetry::TelemetryPolicy;
|
||||
|
||||
// ============= Stats =============
|
||||
|
||||
#[derive(Default)]
|
||||
@@ -30,6 +37,10 @@ pub struct Stats {
|
||||
me_route_drop_no_conn: AtomicU64,
|
||||
me_route_drop_channel_closed: AtomicU64,
|
||||
me_route_drop_queue_full: AtomicU64,
|
||||
me_route_drop_queue_full_base: AtomicU64,
|
||||
me_route_drop_queue_full_high: AtomicU64,
|
||||
me_socks_kdf_strict_reject: AtomicU64,
|
||||
me_socks_kdf_compat_fallback: AtomicU64,
|
||||
secure_padding_invalid: AtomicU64,
|
||||
desync_total: AtomicU64,
|
||||
desync_full_logged: AtomicU64,
|
||||
@@ -42,6 +53,16 @@ pub struct Stats {
|
||||
pool_drain_active: AtomicU64,
|
||||
pool_force_close_total: AtomicU64,
|
||||
pool_stale_pick_total: AtomicU64,
|
||||
me_writer_removed_total: AtomicU64,
|
||||
me_writer_removed_unexpected_total: AtomicU64,
|
||||
me_refill_triggered_total: AtomicU64,
|
||||
me_refill_skipped_inflight_total: AtomicU64,
|
||||
me_refill_failed_total: AtomicU64,
|
||||
me_writer_restored_same_endpoint_total: AtomicU64,
|
||||
me_writer_restored_fallback_total: AtomicU64,
|
||||
telemetry_core_enabled: AtomicBool,
|
||||
telemetry_user_enabled: AtomicBool,
|
||||
telemetry_me_level: AtomicU8,
|
||||
user_stats: DashMap<String, UserStats>,
|
||||
start_time: parking_lot::RwLock<Option<Instant>>,
|
||||
}
|
||||
@@ -59,44 +80,167 @@ pub struct UserStats {
|
||||
impl Stats {
|
||||
pub fn new() -> Self {
|
||||
let stats = Self::default();
|
||||
stats.apply_telemetry_policy(TelemetryPolicy::default());
|
||||
*stats.start_time.write() = Some(Instant::now());
|
||||
stats
|
||||
}
|
||||
|
||||
pub fn increment_connects_all(&self) { self.connects_all.fetch_add(1, Ordering::Relaxed); }
|
||||
pub fn increment_connects_bad(&self) { self.connects_bad.fetch_add(1, Ordering::Relaxed); }
|
||||
pub fn increment_handshake_timeouts(&self) { self.handshake_timeouts.fetch_add(1, Ordering::Relaxed); }
|
||||
pub fn increment_me_keepalive_sent(&self) { self.me_keepalive_sent.fetch_add(1, Ordering::Relaxed); }
|
||||
pub fn increment_me_keepalive_failed(&self) { self.me_keepalive_failed.fetch_add(1, Ordering::Relaxed); }
|
||||
pub fn increment_me_keepalive_pong(&self) { self.me_keepalive_pong.fetch_add(1, Ordering::Relaxed); }
|
||||
pub fn increment_me_keepalive_timeout(&self) { self.me_keepalive_timeout.fetch_add(1, Ordering::Relaxed); }
|
||||
pub fn increment_me_keepalive_timeout_by(&self, value: u64) {
|
||||
self.me_keepalive_timeout.fetch_add(value, Ordering::Relaxed);
|
||||
|
||||
fn telemetry_me_level(&self) -> MeTelemetryLevel {
|
||||
MeTelemetryLevel::from_u8(self.telemetry_me_level.load(Ordering::Relaxed))
|
||||
}
|
||||
|
||||
fn telemetry_core_enabled(&self) -> bool {
|
||||
self.telemetry_core_enabled.load(Ordering::Relaxed)
|
||||
}
|
||||
|
||||
fn telemetry_user_enabled(&self) -> bool {
|
||||
self.telemetry_user_enabled.load(Ordering::Relaxed)
|
||||
}
|
||||
|
||||
fn telemetry_me_allows_normal(&self) -> bool {
|
||||
self.telemetry_me_level().allows_normal()
|
||||
}
|
||||
|
||||
fn telemetry_me_allows_debug(&self) -> bool {
|
||||
self.telemetry_me_level().allows_debug()
|
||||
}
|
||||
|
||||
pub fn apply_telemetry_policy(&self, policy: TelemetryPolicy) {
|
||||
self.telemetry_core_enabled
|
||||
.store(policy.core_enabled, Ordering::Relaxed);
|
||||
self.telemetry_user_enabled
|
||||
.store(policy.user_enabled, Ordering::Relaxed);
|
||||
self.telemetry_me_level
|
||||
.store(policy.me_level.as_u8(), Ordering::Relaxed);
|
||||
}
|
||||
|
||||
pub fn telemetry_policy(&self) -> TelemetryPolicy {
|
||||
TelemetryPolicy {
|
||||
core_enabled: self.telemetry_core_enabled(),
|
||||
user_enabled: self.telemetry_user_enabled(),
|
||||
me_level: self.telemetry_me_level(),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn increment_connects_all(&self) {
|
||||
if self.telemetry_core_enabled() {
|
||||
self.connects_all.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_connects_bad(&self) {
|
||||
if self.telemetry_core_enabled() {
|
||||
self.connects_bad.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_handshake_timeouts(&self) {
|
||||
if self.telemetry_core_enabled() {
|
||||
self.handshake_timeouts.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_me_keepalive_sent(&self) {
|
||||
if self.telemetry_me_allows_debug() {
|
||||
self.me_keepalive_sent.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_me_keepalive_failed(&self) {
|
||||
if self.telemetry_me_allows_normal() {
|
||||
self.me_keepalive_failed.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_me_keepalive_pong(&self) {
|
||||
if self.telemetry_me_allows_debug() {
|
||||
self.me_keepalive_pong.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_me_keepalive_timeout(&self) {
|
||||
if self.telemetry_me_allows_normal() {
|
||||
self.me_keepalive_timeout.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_me_keepalive_timeout_by(&self, value: u64) {
|
||||
if self.telemetry_me_allows_normal() {
|
||||
self.me_keepalive_timeout.fetch_add(value, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_me_reconnect_attempt(&self) {
|
||||
if self.telemetry_me_allows_normal() {
|
||||
self.me_reconnect_attempts.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_me_reconnect_success(&self) {
|
||||
if self.telemetry_me_allows_normal() {
|
||||
self.me_reconnect_success.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_me_crc_mismatch(&self) {
|
||||
if self.telemetry_me_allows_normal() {
|
||||
self.me_crc_mismatch.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_me_seq_mismatch(&self) {
|
||||
if self.telemetry_me_allows_normal() {
|
||||
self.me_seq_mismatch.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_me_route_drop_no_conn(&self) {
|
||||
if self.telemetry_me_allows_normal() {
|
||||
self.me_route_drop_no_conn.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_me_reconnect_attempt(&self) { self.me_reconnect_attempts.fetch_add(1, Ordering::Relaxed); }
|
||||
pub fn increment_me_reconnect_success(&self) { self.me_reconnect_success.fetch_add(1, Ordering::Relaxed); }
|
||||
pub fn increment_me_crc_mismatch(&self) { self.me_crc_mismatch.fetch_add(1, Ordering::Relaxed); }
|
||||
pub fn increment_me_seq_mismatch(&self) { self.me_seq_mismatch.fetch_add(1, Ordering::Relaxed); }
|
||||
pub fn increment_me_route_drop_no_conn(&self) { self.me_route_drop_no_conn.fetch_add(1, Ordering::Relaxed); }
|
||||
pub fn increment_me_route_drop_channel_closed(&self) {
|
||||
self.me_route_drop_channel_closed.fetch_add(1, Ordering::Relaxed);
|
||||
if self.telemetry_me_allows_normal() {
|
||||
self.me_route_drop_channel_closed.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_me_route_drop_queue_full(&self) {
|
||||
self.me_route_drop_queue_full.fetch_add(1, Ordering::Relaxed);
|
||||
if self.telemetry_me_allows_normal() {
|
||||
self.me_route_drop_queue_full.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_me_route_drop_queue_full_base(&self) {
|
||||
if self.telemetry_me_allows_normal() {
|
||||
self.me_route_drop_queue_full_base.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_me_route_drop_queue_full_high(&self) {
|
||||
if self.telemetry_me_allows_normal() {
|
||||
self.me_route_drop_queue_full_high.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_me_socks_kdf_strict_reject(&self) {
|
||||
if self.telemetry_me_allows_normal() {
|
||||
self.me_socks_kdf_strict_reject.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_me_socks_kdf_compat_fallback(&self) {
|
||||
if self.telemetry_me_allows_debug() {
|
||||
self.me_socks_kdf_compat_fallback.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_secure_padding_invalid(&self) {
|
||||
self.secure_padding_invalid.fetch_add(1, Ordering::Relaxed);
|
||||
if self.telemetry_me_allows_normal() {
|
||||
self.secure_padding_invalid.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_desync_total(&self) {
|
||||
self.desync_total.fetch_add(1, Ordering::Relaxed);
|
||||
if self.telemetry_me_allows_normal() {
|
||||
self.desync_total.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_desync_full_logged(&self) {
|
||||
self.desync_full_logged.fetch_add(1, Ordering::Relaxed);
|
||||
if self.telemetry_me_allows_normal() {
|
||||
self.desync_full_logged.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_desync_suppressed(&self) {
|
||||
self.desync_suppressed.fetch_add(1, Ordering::Relaxed);
|
||||
if self.telemetry_me_allows_normal() {
|
||||
self.desync_suppressed.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn observe_desync_frames_ok(&self, frames_ok: u64) {
|
||||
if !self.telemetry_me_allows_normal() {
|
||||
return;
|
||||
}
|
||||
match frames_ok {
|
||||
0 => {
|
||||
self.desync_frames_bucket_0.fetch_add(1, Ordering::Relaxed);
|
||||
@@ -113,12 +257,19 @@ impl Stats {
|
||||
}
|
||||
}
|
||||
pub fn increment_pool_swap_total(&self) {
|
||||
self.pool_swap_total.fetch_add(1, Ordering::Relaxed);
|
||||
if self.telemetry_me_allows_debug() {
|
||||
self.pool_swap_total.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_pool_drain_active(&self) {
|
||||
self.pool_drain_active.fetch_add(1, Ordering::Relaxed);
|
||||
if self.telemetry_me_allows_debug() {
|
||||
self.pool_drain_active.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn decrement_pool_drain_active(&self) {
|
||||
if !self.telemetry_me_allows_debug() {
|
||||
return;
|
||||
}
|
||||
let mut current = self.pool_drain_active.load(Ordering::Relaxed);
|
||||
loop {
|
||||
if current == 0 {
|
||||
@@ -136,10 +287,51 @@ impl Stats {
|
||||
}
|
||||
}
|
||||
pub fn increment_pool_force_close_total(&self) {
|
||||
self.pool_force_close_total.fetch_add(1, Ordering::Relaxed);
|
||||
if self.telemetry_me_allows_normal() {
|
||||
self.pool_force_close_total.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_pool_stale_pick_total(&self) {
|
||||
self.pool_stale_pick_total.fetch_add(1, Ordering::Relaxed);
|
||||
if self.telemetry_me_allows_normal() {
|
||||
self.pool_stale_pick_total.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_me_writer_removed_total(&self) {
|
||||
if self.telemetry_me_allows_debug() {
|
||||
self.me_writer_removed_total.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_me_writer_removed_unexpected_total(&self) {
|
||||
if self.telemetry_me_allows_normal() {
|
||||
self.me_writer_removed_unexpected_total.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_me_refill_triggered_total(&self) {
|
||||
if self.telemetry_me_allows_debug() {
|
||||
self.me_refill_triggered_total.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_me_refill_skipped_inflight_total(&self) {
|
||||
if self.telemetry_me_allows_debug() {
|
||||
self.me_refill_skipped_inflight_total.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_me_refill_failed_total(&self) {
|
||||
if self.telemetry_me_allows_normal() {
|
||||
self.me_refill_failed_total.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_me_writer_restored_same_endpoint_total(&self) {
|
||||
if self.telemetry_me_allows_normal() {
|
||||
self.me_writer_restored_same_endpoint_total
|
||||
.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn increment_me_writer_restored_fallback_total(&self) {
|
||||
if self.telemetry_me_allows_normal() {
|
||||
self.me_writer_restored_fallback_total
|
||||
.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
pub fn get_connects_all(&self) -> u64 { self.connects_all.load(Ordering::Relaxed) }
|
||||
pub fn get_connects_bad(&self) -> u64 { self.connects_bad.load(Ordering::Relaxed) }
|
||||
@@ -158,6 +350,18 @@ impl Stats {
|
||||
pub fn get_me_route_drop_queue_full(&self) -> u64 {
|
||||
self.me_route_drop_queue_full.load(Ordering::Relaxed)
|
||||
}
|
||||
pub fn get_me_route_drop_queue_full_base(&self) -> u64 {
|
||||
self.me_route_drop_queue_full_base.load(Ordering::Relaxed)
|
||||
}
|
||||
pub fn get_me_route_drop_queue_full_high(&self) -> u64 {
|
||||
self.me_route_drop_queue_full_high.load(Ordering::Relaxed)
|
||||
}
|
||||
pub fn get_me_socks_kdf_strict_reject(&self) -> u64 {
|
||||
self.me_socks_kdf_strict_reject.load(Ordering::Relaxed)
|
||||
}
|
||||
pub fn get_me_socks_kdf_compat_fallback(&self) -> u64 {
|
||||
self.me_socks_kdf_compat_fallback.load(Ordering::Relaxed)
|
||||
}
|
||||
pub fn get_secure_padding_invalid(&self) -> u64 {
|
||||
self.secure_padding_invalid.load(Ordering::Relaxed)
|
||||
}
|
||||
@@ -194,13 +398,40 @@ impl Stats {
|
||||
pub fn get_pool_stale_pick_total(&self) -> u64 {
|
||||
self.pool_stale_pick_total.load(Ordering::Relaxed)
|
||||
}
|
||||
pub fn get_me_writer_removed_total(&self) -> u64 {
|
||||
self.me_writer_removed_total.load(Ordering::Relaxed)
|
||||
}
|
||||
pub fn get_me_writer_removed_unexpected_total(&self) -> u64 {
|
||||
self.me_writer_removed_unexpected_total.load(Ordering::Relaxed)
|
||||
}
|
||||
pub fn get_me_refill_triggered_total(&self) -> u64 {
|
||||
self.me_refill_triggered_total.load(Ordering::Relaxed)
|
||||
}
|
||||
pub fn get_me_refill_skipped_inflight_total(&self) -> u64 {
|
||||
self.me_refill_skipped_inflight_total.load(Ordering::Relaxed)
|
||||
}
|
||||
pub fn get_me_refill_failed_total(&self) -> u64 {
|
||||
self.me_refill_failed_total.load(Ordering::Relaxed)
|
||||
}
|
||||
pub fn get_me_writer_restored_same_endpoint_total(&self) -> u64 {
|
||||
self.me_writer_restored_same_endpoint_total.load(Ordering::Relaxed)
|
||||
}
|
||||
pub fn get_me_writer_restored_fallback_total(&self) -> u64 {
|
||||
self.me_writer_restored_fallback_total.load(Ordering::Relaxed)
|
||||
}
|
||||
|
||||
pub fn increment_user_connects(&self, user: &str) {
|
||||
if !self.telemetry_user_enabled() {
|
||||
return;
|
||||
}
|
||||
self.user_stats.entry(user.to_string()).or_default()
|
||||
.connects.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
pub fn increment_user_curr_connects(&self, user: &str) {
|
||||
if !self.telemetry_user_enabled() {
|
||||
return;
|
||||
}
|
||||
self.user_stats.entry(user.to_string()).or_default()
|
||||
.curr_connects.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
@@ -233,21 +464,33 @@ impl Stats {
|
||||
}
|
||||
|
||||
pub fn add_user_octets_from(&self, user: &str, bytes: u64) {
|
||||
if !self.telemetry_user_enabled() {
|
||||
return;
|
||||
}
|
||||
self.user_stats.entry(user.to_string()).or_default()
|
||||
.octets_from_client.fetch_add(bytes, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
pub fn add_user_octets_to(&self, user: &str, bytes: u64) {
|
||||
if !self.telemetry_user_enabled() {
|
||||
return;
|
||||
}
|
||||
self.user_stats.entry(user.to_string()).or_default()
|
||||
.octets_to_client.fetch_add(bytes, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
pub fn increment_user_msgs_from(&self, user: &str) {
|
||||
if !self.telemetry_user_enabled() {
|
||||
return;
|
||||
}
|
||||
self.user_stats.entry(user.to_string()).or_default()
|
||||
.msgs_from_client.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
pub fn increment_user_msgs_to(&self, user: &str) {
|
||||
if !self.telemetry_user_enabled() {
|
||||
return;
|
||||
}
|
||||
self.user_stats.entry(user.to_string()).or_default()
|
||||
.msgs_to_client.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
@@ -325,10 +568,10 @@ impl ReplayShard {
|
||||
|
||||
// Use key.as_ref() to get &[u8] — avoids Borrow<Q> ambiguity
|
||||
// between Borrow<[u8]> and Borrow<Box<[u8]>>
|
||||
if let Some(entry) = self.cache.peek(key.as_ref()) {
|
||||
if entry.seq == queue_seq {
|
||||
self.cache.pop(key.as_ref());
|
||||
}
|
||||
if let Some(entry) = self.cache.peek(key.as_ref())
|
||||
&& entry.seq == queue_seq
|
||||
{
|
||||
self.cache.pop(key.as_ref());
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -496,6 +739,8 @@ impl ReplayStats {
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::config::MeTelemetryLevel;
|
||||
use std::sync::Arc;
|
||||
|
||||
#[test]
|
||||
fn test_stats_shared_counters() {
|
||||
@@ -505,6 +750,40 @@ mod tests {
|
||||
stats.increment_connects_all();
|
||||
assert_eq!(stats.get_connects_all(), 3);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_telemetry_policy_disables_core_and_user_counters() {
|
||||
let stats = Stats::new();
|
||||
stats.apply_telemetry_policy(TelemetryPolicy {
|
||||
core_enabled: false,
|
||||
user_enabled: false,
|
||||
me_level: MeTelemetryLevel::Normal,
|
||||
});
|
||||
|
||||
stats.increment_connects_all();
|
||||
stats.increment_user_connects("alice");
|
||||
stats.add_user_octets_from("alice", 1024);
|
||||
assert_eq!(stats.get_connects_all(), 0);
|
||||
assert_eq!(stats.get_user_curr_connects("alice"), 0);
|
||||
assert_eq!(stats.get_user_total_octets("alice"), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_telemetry_policy_me_silent_blocks_me_counters() {
|
||||
let stats = Stats::new();
|
||||
stats.apply_telemetry_policy(TelemetryPolicy {
|
||||
core_enabled: true,
|
||||
user_enabled: true,
|
||||
me_level: MeTelemetryLevel::Silent,
|
||||
});
|
||||
|
||||
stats.increment_me_crc_mismatch();
|
||||
stats.increment_me_keepalive_sent();
|
||||
stats.increment_me_route_drop_queue_full();
|
||||
assert_eq!(stats.get_me_crc_mismatch(), 0);
|
||||
assert_eq!(stats.get_me_keepalive_sent(), 0);
|
||||
assert_eq!(stats.get_me_route_drop_queue_full(), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_replay_checker_basic() {
|
||||
|
||||
29
src/stats/telemetry.rs
Normal file
29
src/stats/telemetry.rs
Normal file
@@ -0,0 +1,29 @@
|
||||
use crate::config::{MeTelemetryLevel, TelemetryConfig};
|
||||
|
||||
/// Runtime telemetry policy used by hot-path counters.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub struct TelemetryPolicy {
|
||||
pub core_enabled: bool,
|
||||
pub user_enabled: bool,
|
||||
pub me_level: MeTelemetryLevel,
|
||||
}
|
||||
|
||||
impl Default for TelemetryPolicy {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
core_enabled: true,
|
||||
user_enabled: true,
|
||||
me_level: MeTelemetryLevel::Normal,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl TelemetryPolicy {
|
||||
pub fn from_config(cfg: &TelemetryConfig) -> Self {
|
||||
Self {
|
||||
core_enabled: cfg.core_enabled,
|
||||
user_enabled: cfg.user_enabled,
|
||||
me_level: cfg.me_level,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -3,6 +3,8 @@
|
||||
//! This module provides a thread-safe pool of BytesMut buffers
|
||||
//! that can be reused across connections to reduce allocation pressure.
|
||||
|
||||
#![allow(dead_code)]
|
||||
|
||||
use bytes::BytesMut;
|
||||
use crossbeam_queue::ArrayQueue;
|
||||
use std::ops::{Deref, DerefMut};
|
||||
|
||||
@@ -18,6 +18,8 @@
|
||||
//! is either written to upstream or stored in our pending buffer
|
||||
//! - when upstream is pending -> ciphertext is buffered/bounded and backpressure is applied
|
||||
//!
|
||||
|
||||
#![allow(dead_code)]
|
||||
//! =======================
|
||||
//! Writer state machine
|
||||
//! =======================
|
||||
@@ -45,7 +47,7 @@
|
||||
//! - when upstream is Pending but pending still has room: accept `to_accept` bytes and
|
||||
//! encrypt+append ciphertext directly into pending (in-place encryption of appended range)
|
||||
|
||||
//! Encrypted stream wrappers using AES-CTR
|
||||
//! Encrypted stream wrappers using AES-CTR
|
||||
//!
|
||||
//! This module provides stateful async stream wrappers that handle
|
||||
//! encryption/decryption with proper partial read/write handling.
|
||||
@@ -55,7 +57,7 @@ use std::io::{self, ErrorKind, Result};
|
||||
use std::pin::Pin;
|
||||
use std::task::{Context, Poll};
|
||||
use tokio::io::{AsyncRead, AsyncWrite, ReadBuf};
|
||||
use tracing::{debug, trace, warn};
|
||||
use tracing::{debug, trace};
|
||||
|
||||
use crate::crypto::AesCtr;
|
||||
use super::state::{StreamState, YieldBuffer};
|
||||
@@ -151,9 +153,9 @@ impl<R> CryptoReader<R> {
|
||||
fn take_poison_error(&mut self) -> io::Error {
|
||||
match &mut self.state {
|
||||
CryptoReaderState::Poisoned { error } => error.take().unwrap_or_else(|| {
|
||||
io::Error::new(ErrorKind::Other, "stream previously poisoned")
|
||||
io::Error::other("stream previously poisoned")
|
||||
}),
|
||||
_ => io::Error::new(ErrorKind::Other, "stream not poisoned"),
|
||||
_ => io::Error::other("stream not poisoned"),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -166,6 +168,7 @@ impl<R: AsyncRead + Unpin> AsyncRead for CryptoReader<R> {
|
||||
) -> Poll<Result<()>> {
|
||||
let this = self.get_mut();
|
||||
|
||||
#[allow(clippy::never_loop)]
|
||||
loop {
|
||||
match &mut this.state {
|
||||
CryptoReaderState::Poisoned { .. } => {
|
||||
@@ -483,14 +486,14 @@ impl<W> CryptoWriter<W> {
|
||||
fn take_poison_error(&mut self) -> io::Error {
|
||||
match &mut self.state {
|
||||
CryptoWriterState::Poisoned { error } => error.take().unwrap_or_else(|| {
|
||||
io::Error::new(ErrorKind::Other, "stream previously poisoned")
|
||||
io::Error::other("stream previously poisoned")
|
||||
}),
|
||||
_ => io::Error::new(ErrorKind::Other, "stream not poisoned"),
|
||||
_ => io::Error::other("stream not poisoned"),
|
||||
}
|
||||
}
|
||||
|
||||
/// Ensure we are in Flushing state and return mutable pending buffer.
|
||||
fn ensure_pending<'a>(state: &'a mut CryptoWriterState, max_pending: usize) -> &'a mut PendingCiphertext {
|
||||
fn ensure_pending(state: &mut CryptoWriterState, max_pending: usize) -> &mut PendingCiphertext {
|
||||
if matches!(state, CryptoWriterState::Idle) {
|
||||
*state = CryptoWriterState::Flushing {
|
||||
pending: PendingCiphertext::new(max_pending),
|
||||
|
||||
@@ -3,6 +3,8 @@
|
||||
//! This module defines the common types and traits used by all
|
||||
//! frame encoding/decoding implementations.
|
||||
|
||||
#![allow(dead_code)]
|
||||
|
||||
use bytes::{Bytes, BytesMut};
|
||||
use std::io::Result;
|
||||
use std::sync::Arc;
|
||||
|
||||
@@ -3,6 +3,8 @@
|
||||
//! This module provides Encoder/Decoder implementations compatible
|
||||
//! with tokio-util's Framed wrapper for easy async frame I/O.
|
||||
|
||||
#![allow(dead_code)]
|
||||
|
||||
use bytes::{Bytes, BytesMut, BufMut};
|
||||
use std::io::{self, Error, ErrorKind};
|
||||
use std::sync::Arc;
|
||||
@@ -137,7 +139,7 @@ fn encode_abridged(frame: &Frame, dst: &mut BytesMut) -> io::Result<()> {
|
||||
let data = &frame.data;
|
||||
|
||||
// Validate alignment
|
||||
if data.len() % 4 != 0 {
|
||||
if !data.len().is_multiple_of(4) {
|
||||
return Err(Error::new(
|
||||
ErrorKind::InvalidInput,
|
||||
format!("abridged frame must be 4-byte aligned, got {} bytes", data.len())
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
//! MTProto frame stream wrappers
|
||||
|
||||
use bytes::{Bytes, BytesMut};
|
||||
#![allow(dead_code)]
|
||||
|
||||
use bytes::Bytes;
|
||||
use std::io::{Error, ErrorKind, Result};
|
||||
use tokio::io::{AsyncRead, AsyncWrite, AsyncReadExt, AsyncWriteExt};
|
||||
use crate::protocol::constants::*;
|
||||
@@ -76,7 +78,7 @@ impl<W> AbridgedFrameWriter<W> {
|
||||
impl<W: AsyncWrite + Unpin> AbridgedFrameWriter<W> {
|
||||
/// Write a frame
|
||||
pub async fn write_frame(&mut self, data: &[u8], meta: &FrameMeta) -> Result<()> {
|
||||
if data.len() % 4 != 0 {
|
||||
if !data.len().is_multiple_of(4) {
|
||||
return Err(Error::new(
|
||||
ErrorKind::InvalidInput,
|
||||
format!("Abridged frame must be aligned to 4 bytes, got {}", data.len()),
|
||||
@@ -329,7 +331,7 @@ impl<R: AsyncRead + Unpin> MtprotoFrameReader<R> {
|
||||
}
|
||||
|
||||
// Validate length
|
||||
if len < MIN_MSG_LEN || len > MAX_MSG_LEN || len % PADDING_FILLER.len() != 0 {
|
||||
if !(MIN_MSG_LEN..=MAX_MSG_LEN).contains(&len) || !len.is_multiple_of(PADDING_FILLER.len()) {
|
||||
return Err(Error::new(
|
||||
ErrorKind::InvalidData,
|
||||
format!("Invalid message length: {}", len),
|
||||
|
||||
@@ -12,28 +12,34 @@ pub mod frame_codec;
|
||||
pub mod frame_stream;
|
||||
|
||||
// Re-export state machine types
|
||||
#[allow(unused_imports)]
|
||||
pub use state::{
|
||||
StreamState, Transition, PollResult,
|
||||
ReadBuffer, WriteBuffer, HeaderBuffer, YieldBuffer,
|
||||
};
|
||||
|
||||
// Re-export buffer pool
|
||||
#[allow(unused_imports)]
|
||||
pub use buffer_pool::{BufferPool, PooledBuffer, PoolStats};
|
||||
|
||||
// Re-export stream implementations
|
||||
#[allow(unused_imports)]
|
||||
pub use crypto_stream::{CryptoReader, CryptoWriter, PassthroughStream};
|
||||
pub use tls_stream::{FakeTlsReader, FakeTlsWriter};
|
||||
|
||||
// Re-export frame types
|
||||
#[allow(unused_imports)]
|
||||
pub use frame::{Frame, FrameMeta, FrameCodec as FrameCodecTrait, create_codec};
|
||||
|
||||
// Re-export tokio-util compatible codecs
|
||||
// Re-export tokio-util compatible codecs
|
||||
#[allow(unused_imports)]
|
||||
pub use frame_codec::{
|
||||
FrameCodec,
|
||||
AbridgedCodec, IntermediateCodec, SecureCodec,
|
||||
};
|
||||
|
||||
// Legacy re-exports for compatibility
|
||||
#[allow(unused_imports)]
|
||||
pub use frame_stream::{
|
||||
AbridgedFrameReader, AbridgedFrameWriter,
|
||||
IntermediateFrameReader, IntermediateFrameWriter,
|
||||
|
||||
@@ -3,6 +3,8 @@
|
||||
//! This module provides core types and traits for implementing
|
||||
//! stateful async streams with proper partial read/write handling.
|
||||
|
||||
#![allow(dead_code)]
|
||||
|
||||
use bytes::{Bytes, BytesMut};
|
||||
use std::io;
|
||||
|
||||
|
||||
@@ -18,6 +18,8 @@
|
||||
//! - Explicit state machines for all async operations
|
||||
//! - Never lose data on partial reads
|
||||
//! - Atomic TLS record formation for writes
|
||||
|
||||
#![allow(dead_code)]
|
||||
//! - Proper handling of all TLS record types
|
||||
//!
|
||||
//! Important nuance (Telegram FakeTLS):
|
||||
@@ -133,7 +135,7 @@ impl TlsRecordHeader {
|
||||
}
|
||||
|
||||
/// Build header bytes
|
||||
fn to_bytes(&self) -> [u8; 5] {
|
||||
fn to_bytes(self) -> [u8; 5] {
|
||||
[
|
||||
self.record_type,
|
||||
self.version[0],
|
||||
@@ -258,9 +260,9 @@ impl<R> FakeTlsReader<R> {
|
||||
fn take_poison_error(&mut self) -> io::Error {
|
||||
match &mut self.state {
|
||||
TlsReaderState::Poisoned { error } => error.take().unwrap_or_else(|| {
|
||||
io::Error::new(ErrorKind::Other, "stream previously poisoned")
|
||||
io::Error::other("stream previously poisoned")
|
||||
}),
|
||||
_ => io::Error::new(ErrorKind::Other, "stream not poisoned"),
|
||||
_ => io::Error::other("stream not poisoned"),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -295,7 +297,7 @@ impl<R: AsyncRead + Unpin> AsyncRead for FakeTlsReader<R> {
|
||||
TlsReaderState::Poisoned { error } => {
|
||||
this.state = TlsReaderState::Poisoned { error: None };
|
||||
let err = error.unwrap_or_else(|| {
|
||||
io::Error::new(ErrorKind::Other, "stream previously poisoned")
|
||||
io::Error::other("stream previously poisoned")
|
||||
});
|
||||
return Poll::Ready(Err(err));
|
||||
}
|
||||
@@ -614,9 +616,9 @@ impl<W> FakeTlsWriter<W> {
|
||||
fn take_poison_error(&mut self) -> io::Error {
|
||||
match &mut self.state {
|
||||
TlsWriterState::Poisoned { error } => error.take().unwrap_or_else(|| {
|
||||
io::Error::new(ErrorKind::Other, "stream previously poisoned")
|
||||
io::Error::other("stream previously poisoned")
|
||||
}),
|
||||
_ => io::Error::new(ErrorKind::Other, "stream not poisoned"),
|
||||
_ => io::Error::other("stream not poisoned"),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -680,7 +682,7 @@ impl<W: AsyncWrite + Unpin> AsyncWrite for FakeTlsWriter<W> {
|
||||
TlsWriterState::Poisoned { error } => {
|
||||
this.state = TlsWriterState::Poisoned { error: None };
|
||||
let err = error.unwrap_or_else(|| {
|
||||
Error::new(ErrorKind::Other, "stream previously poisoned")
|
||||
Error::other("stream previously poisoned")
|
||||
});
|
||||
return Poll::Ready(Err(err));
|
||||
}
|
||||
@@ -769,7 +771,7 @@ impl<W: AsyncWrite + Unpin> AsyncWrite for FakeTlsWriter<W> {
|
||||
TlsWriterState::Poisoned { error } => {
|
||||
this.state = TlsWriterState::Poisoned { error: None };
|
||||
let err = error.unwrap_or_else(|| {
|
||||
Error::new(ErrorKind::Other, "stream previously poisoned")
|
||||
Error::other("stream previously poisoned")
|
||||
});
|
||||
return Poll::Ready(Err(err));
|
||||
}
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
//! Stream traits and common types
|
||||
|
||||
#![allow(dead_code)]
|
||||
|
||||
use bytes::Bytes;
|
||||
use std::io::Result;
|
||||
use std::pin::Pin;
|
||||
|
||||
@@ -19,6 +19,7 @@ pub struct TlsFrontCache {
|
||||
disk_path: PathBuf,
|
||||
}
|
||||
|
||||
#[allow(dead_code)]
|
||||
impl TlsFrontCache {
|
||||
pub fn new(domains: &[String], default_len: usize, disk_path: impl AsRef<Path>) -> Self {
|
||||
let default_template = ParsedServerHello {
|
||||
@@ -114,32 +115,32 @@ impl TlsFrontCache {
|
||||
if !name.ends_with(".json") {
|
||||
continue;
|
||||
}
|
||||
if let Ok(data) = tokio::fs::read(entry.path()).await {
|
||||
if let Ok(mut cached) = serde_json::from_slice::<CachedTlsData>(&data) {
|
||||
if cached.domain.is_empty()
|
||||
|| cached.domain.len() > 255
|
||||
|| !cached.domain.chars().all(|c| c.is_ascii_alphanumeric() || c == '.' || c == '-')
|
||||
{
|
||||
warn!(file = %name, "Skipping TLS cache entry with invalid domain");
|
||||
continue;
|
||||
}
|
||||
// fetched_at is skipped during deserialization; approximate with file mtime if available.
|
||||
if let Ok(meta) = entry.metadata().await {
|
||||
if let Ok(modified) = meta.modified() {
|
||||
cached.fetched_at = modified;
|
||||
}
|
||||
}
|
||||
// Drop entries older than 72h
|
||||
if let Ok(age) = cached.fetched_at.elapsed() {
|
||||
if age > Duration::from_secs(72 * 3600) {
|
||||
warn!(domain = %cached.domain, "Skipping stale TLS cache entry (>72h)");
|
||||
continue;
|
||||
}
|
||||
}
|
||||
let domain = cached.domain.clone();
|
||||
self.set(&domain, cached).await;
|
||||
loaded += 1;
|
||||
if let Ok(data) = tokio::fs::read(entry.path()).await
|
||||
&& let Ok(mut cached) = serde_json::from_slice::<CachedTlsData>(&data)
|
||||
{
|
||||
if cached.domain.is_empty()
|
||||
|| cached.domain.len() > 255
|
||||
|| !cached.domain.chars().all(|c| c.is_ascii_alphanumeric() || c == '.' || c == '-')
|
||||
{
|
||||
warn!(file = %name, "Skipping TLS cache entry with invalid domain");
|
||||
continue;
|
||||
}
|
||||
// fetched_at is skipped during deserialization; approximate with file mtime if available.
|
||||
if let Ok(meta) = entry.metadata().await
|
||||
&& let Ok(modified) = meta.modified()
|
||||
{
|
||||
cached.fetched_at = modified;
|
||||
}
|
||||
// Drop entries older than 72h
|
||||
if let Ok(age) = cached.fetched_at.elapsed()
|
||||
&& age > Duration::from_secs(72 * 3600)
|
||||
{
|
||||
warn!(domain = %cached.domain, "Skipping stale TLS cache entry (>72h)");
|
||||
continue;
|
||||
}
|
||||
let domain = cached.domain.clone();
|
||||
self.set(&domain, cached).await;
|
||||
loaded += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -173,7 +174,7 @@ impl TlsFrontCache {
|
||||
tokio::spawn(async move {
|
||||
loop {
|
||||
for domain in &domains {
|
||||
fetcher(domain.clone()).await;
|
||||
let _ = fetcher(domain.clone()).await;
|
||||
}
|
||||
sleep(interval).await;
|
||||
}
|
||||
|
||||
@@ -12,7 +12,7 @@ fn jitter_and_clamp_sizes(sizes: &[usize], rng: &SecureRandom) -> Vec<usize> {
|
||||
sizes
|
||||
.iter()
|
||||
.map(|&size| {
|
||||
let base = size.max(MIN_APP_DATA).min(MAX_APP_DATA);
|
||||
let base = size.clamp(MIN_APP_DATA, MAX_APP_DATA);
|
||||
let jitter_range = ((base as f64) * 0.03).round() as i64;
|
||||
if jitter_range == 0 {
|
||||
return base;
|
||||
@@ -50,7 +50,7 @@ fn ensure_payload_capacity(mut sizes: Vec<usize>, payload_len: usize) -> Vec<usi
|
||||
|
||||
while body_total < payload_len {
|
||||
let remaining = payload_len - body_total;
|
||||
let chunk = (remaining + 17).min(MAX_APP_DATA).max(MIN_APP_DATA);
|
||||
let chunk = (remaining + 17).clamp(MIN_APP_DATA, MAX_APP_DATA);
|
||||
sizes.push(chunk);
|
||||
body_total += chunk.saturating_sub(17);
|
||||
}
|
||||
@@ -189,7 +189,7 @@ pub fn build_emulated_server_hello(
|
||||
.as_ref()
|
||||
.map(|payload| payload.certificate_message.as_slice())
|
||||
.filter(|payload| !payload.is_empty())
|
||||
.or_else(|| compact_payload.as_deref())
|
||||
.or(compact_payload.as_deref())
|
||||
} else {
|
||||
compact_payload.as_deref()
|
||||
};
|
||||
@@ -223,15 +223,13 @@ pub fn build_emulated_server_hello(
|
||||
} else {
|
||||
rec.extend_from_slice(&rng.bytes(size));
|
||||
}
|
||||
} else if size > 17 {
|
||||
let body_len = size - 17;
|
||||
rec.extend_from_slice(&rng.bytes(body_len));
|
||||
rec.push(0x16); // inner content type marker (handshake)
|
||||
rec.extend_from_slice(&rng.bytes(16)); // AEAD-like tag
|
||||
} else {
|
||||
if size > 17 {
|
||||
let body_len = size - 17;
|
||||
rec.extend_from_slice(&rng.bytes(body_len));
|
||||
rec.push(0x16); // inner content type marker (handshake)
|
||||
rec.extend_from_slice(&rng.bytes(16)); // AEAD-like tag
|
||||
} else {
|
||||
rec.extend_from_slice(&rng.bytes(size));
|
||||
}
|
||||
rec.extend_from_slice(&rng.bytes(size));
|
||||
}
|
||||
app_data.extend_from_slice(&rec);
|
||||
}
|
||||
|
||||
@@ -2,8 +2,10 @@ use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
use anyhow::{Result, anyhow};
|
||||
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
||||
use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt};
|
||||
use tokio::net::TcpStream;
|
||||
#[cfg(unix)]
|
||||
use tokio::net::UnixStream;
|
||||
use tokio::time::timeout;
|
||||
use tokio_rustls::client::TlsStream;
|
||||
use tokio_rustls::TlsConnector;
|
||||
@@ -18,7 +20,9 @@ use x509_parser::prelude::FromDer;
|
||||
use x509_parser::certificate::X509Certificate;
|
||||
|
||||
use crate::crypto::SecureRandom;
|
||||
use crate::network::dns_overrides::resolve_socket_addr;
|
||||
use crate::protocol::constants::{TLS_RECORD_APPLICATION, TLS_RECORD_HANDSHAKE};
|
||||
use crate::transport::proxy_protocol::{ProxyProtocolV1Builder, ProxyProtocolV2Builder};
|
||||
use crate::tls_front::types::{
|
||||
ParsedCertificateInfo,
|
||||
ParsedServerHello,
|
||||
@@ -210,7 +214,10 @@ fn gen_key_share(rng: &SecureRandom) -> [u8; 32] {
|
||||
key
|
||||
}
|
||||
|
||||
async fn read_tls_record(stream: &mut TcpStream) -> Result<(u8, Vec<u8>)> {
|
||||
async fn read_tls_record<S>(stream: &mut S) -> Result<(u8, Vec<u8>)>
|
||||
where
|
||||
S: AsyncRead + Unpin,
|
||||
{
|
||||
let mut header = [0u8; 5];
|
||||
stream.read_exact(&mut header).await?;
|
||||
let len = u16::from_be_bytes([header[3], header[4]]) as usize;
|
||||
@@ -332,6 +339,55 @@ fn u24_bytes(value: usize) -> Option<[u8; 3]> {
|
||||
])
|
||||
}
|
||||
|
||||
async fn connect_with_dns_override(
|
||||
host: &str,
|
||||
port: u16,
|
||||
connect_timeout: Duration,
|
||||
) -> Result<TcpStream> {
|
||||
if let Some(addr) = resolve_socket_addr(host, port) {
|
||||
return Ok(timeout(connect_timeout, TcpStream::connect(addr)).await??);
|
||||
}
|
||||
Ok(timeout(connect_timeout, TcpStream::connect((host, port))).await??)
|
||||
}
|
||||
|
||||
async fn connect_tcp_with_upstream(
|
||||
host: &str,
|
||||
port: u16,
|
||||
connect_timeout: Duration,
|
||||
upstream: Option<std::sync::Arc<crate::transport::UpstreamManager>>,
|
||||
) -> Result<TcpStream> {
|
||||
if let Some(manager) = upstream {
|
||||
if let Some(addr) = resolve_socket_addr(host, port) {
|
||||
match manager.connect(addr, None, None).await {
|
||||
Ok(stream) => return Ok(stream),
|
||||
Err(e) => {
|
||||
warn!(
|
||||
host = %host,
|
||||
port = port,
|
||||
error = %e,
|
||||
"Upstream connect failed, using direct connect"
|
||||
);
|
||||
}
|
||||
}
|
||||
} else if let Ok(mut addrs) = tokio::net::lookup_host((host, port)).await {
|
||||
if let Some(addr) = addrs.find(|a| a.is_ipv4()) {
|
||||
match manager.connect(addr, None, None).await {
|
||||
Ok(stream) => return Ok(stream),
|
||||
Err(e) => {
|
||||
warn!(
|
||||
host = %host,
|
||||
port = port,
|
||||
error = %e,
|
||||
"Upstream connect failed, using direct connect"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
connect_with_dns_override(host, port, connect_timeout).await
|
||||
}
|
||||
|
||||
fn encode_tls13_certificate_message(cert_chain_der: &[Vec<u8>]) -> Option<Vec<u8>> {
|
||||
if cert_chain_der.is_empty() {
|
||||
return None;
|
||||
@@ -361,18 +417,25 @@ fn encode_tls13_certificate_message(cert_chain_der: &[Vec<u8>]) -> Option<Vec<u8
|
||||
Some(message)
|
||||
}
|
||||
|
||||
async fn fetch_via_raw_tls(
|
||||
host: &str,
|
||||
port: u16,
|
||||
async fn fetch_via_raw_tls_stream<S>(
|
||||
mut stream: S,
|
||||
sni: &str,
|
||||
connect_timeout: Duration,
|
||||
) -> Result<TlsFetchResult> {
|
||||
let addr = format!("{host}:{port}");
|
||||
let mut stream = timeout(connect_timeout, TcpStream::connect(addr)).await??;
|
||||
|
||||
proxy_protocol: u8,
|
||||
) -> Result<TlsFetchResult>
|
||||
where
|
||||
S: AsyncRead + AsyncWrite + Unpin,
|
||||
{
|
||||
let rng = SecureRandom::new();
|
||||
let client_hello = build_client_hello(sni, &rng);
|
||||
timeout(connect_timeout, async {
|
||||
if proxy_protocol > 0 {
|
||||
let header = match proxy_protocol {
|
||||
2 => ProxyProtocolV2Builder::new().build(),
|
||||
_ => ProxyProtocolV1Builder::new().build(),
|
||||
};
|
||||
stream.write_all(&header).await?;
|
||||
}
|
||||
stream.write_all(&client_hello).await?;
|
||||
stream.flush().await?;
|
||||
Ok::<(), std::io::Error>(())
|
||||
@@ -384,7 +447,7 @@ async fn fetch_via_raw_tls(
|
||||
for _ in 0..4 {
|
||||
match timeout(connect_timeout, read_tls_record(&mut stream)).await {
|
||||
Ok(Ok(rec)) => records.push(rec),
|
||||
Ok(Err(e)) => return Err(e.into()),
|
||||
Ok(Err(e)) => return Err(e),
|
||||
Err(_) => break,
|
||||
}
|
||||
if records.len() >= 3 && records.iter().any(|(t, _)| *t == TLS_RECORD_APPLICATION) {
|
||||
@@ -418,34 +481,69 @@ async fn fetch_via_raw_tls(
|
||||
})
|
||||
}
|
||||
|
||||
async fn fetch_via_rustls(
|
||||
async fn fetch_via_raw_tls(
|
||||
host: &str,
|
||||
port: u16,
|
||||
sni: &str,
|
||||
connect_timeout: Duration,
|
||||
upstream: Option<std::sync::Arc<crate::transport::UpstreamManager>>,
|
||||
proxy_protocol: u8,
|
||||
unix_sock: Option<&str>,
|
||||
) -> Result<TlsFetchResult> {
|
||||
// rustls handshake path for certificate and basic negotiated metadata.
|
||||
let stream = if let Some(manager) = upstream {
|
||||
// Resolve host to SocketAddr
|
||||
if let Ok(mut addrs) = tokio::net::lookup_host((host, port)).await {
|
||||
if let Some(addr) = addrs.find(|a| a.is_ipv4()) {
|
||||
match manager.connect(addr, None, None).await {
|
||||
Ok(s) => s,
|
||||
Err(e) => {
|
||||
warn!(sni = %sni, error = %e, "Upstream connect failed, using direct connect");
|
||||
timeout(connect_timeout, TcpStream::connect((host, port))).await??
|
||||
}
|
||||
}
|
||||
} else {
|
||||
timeout(connect_timeout, TcpStream::connect((host, port))).await??
|
||||
#[cfg(unix)]
|
||||
if let Some(sock_path) = unix_sock {
|
||||
match timeout(connect_timeout, UnixStream::connect(sock_path)).await {
|
||||
Ok(Ok(stream)) => {
|
||||
debug!(
|
||||
sni = %sni,
|
||||
sock = %sock_path,
|
||||
"Raw TLS fetch using mask unix socket"
|
||||
);
|
||||
return fetch_via_raw_tls_stream(stream, sni, connect_timeout, 0).await;
|
||||
}
|
||||
Ok(Err(e)) => {
|
||||
warn!(
|
||||
sni = %sni,
|
||||
sock = %sock_path,
|
||||
error = %e,
|
||||
"Raw TLS unix socket connect failed, falling back to TCP"
|
||||
);
|
||||
}
|
||||
Err(_) => {
|
||||
warn!(
|
||||
sni = %sni,
|
||||
sock = %sock_path,
|
||||
"Raw TLS unix socket connect timed out, falling back to TCP"
|
||||
);
|
||||
}
|
||||
} else {
|
||||
timeout(connect_timeout, TcpStream::connect((host, port))).await??
|
||||
}
|
||||
} else {
|
||||
timeout(connect_timeout, TcpStream::connect((host, port))).await??
|
||||
};
|
||||
}
|
||||
|
||||
#[cfg(not(unix))]
|
||||
let _ = unix_sock;
|
||||
|
||||
let stream = connect_tcp_with_upstream(host, port, connect_timeout, upstream).await?;
|
||||
fetch_via_raw_tls_stream(stream, sni, connect_timeout, proxy_protocol).await
|
||||
}
|
||||
|
||||
async fn fetch_via_rustls_stream<S>(
|
||||
mut stream: S,
|
||||
host: &str,
|
||||
sni: &str,
|
||||
proxy_protocol: u8,
|
||||
) -> Result<TlsFetchResult>
|
||||
where
|
||||
S: AsyncRead + AsyncWrite + Unpin,
|
||||
{
|
||||
// rustls handshake path for certificate and basic negotiated metadata.
|
||||
if proxy_protocol > 0 {
|
||||
let header = match proxy_protocol {
|
||||
2 => ProxyProtocolV2Builder::new().build(),
|
||||
_ => ProxyProtocolV1Builder::new().build(),
|
||||
};
|
||||
stream.write_all(&header).await?;
|
||||
stream.flush().await?;
|
||||
}
|
||||
|
||||
let config = build_client_config();
|
||||
let connector = TlsConnector::from(config);
|
||||
@@ -454,7 +552,7 @@ async fn fetch_via_rustls(
|
||||
.or_else(|_| ServerName::try_from(host.to_owned()))
|
||||
.map_err(|_| RustlsError::General("invalid SNI".into()))?;
|
||||
|
||||
let tls_stream: TlsStream<TcpStream> = connector.connect(server_name, stream).await?;
|
||||
let tls_stream: TlsStream<S> = connector.connect(server_name, stream).await?;
|
||||
|
||||
// Extract negotiated parameters and certificates
|
||||
let (_io, session) = tls_stream.get_ref();
|
||||
@@ -515,6 +613,51 @@ async fn fetch_via_rustls(
|
||||
})
|
||||
}
|
||||
|
||||
async fn fetch_via_rustls(
|
||||
host: &str,
|
||||
port: u16,
|
||||
sni: &str,
|
||||
connect_timeout: Duration,
|
||||
upstream: Option<std::sync::Arc<crate::transport::UpstreamManager>>,
|
||||
proxy_protocol: u8,
|
||||
unix_sock: Option<&str>,
|
||||
) -> Result<TlsFetchResult> {
|
||||
#[cfg(unix)]
|
||||
if let Some(sock_path) = unix_sock {
|
||||
match timeout(connect_timeout, UnixStream::connect(sock_path)).await {
|
||||
Ok(Ok(stream)) => {
|
||||
debug!(
|
||||
sni = %sni,
|
||||
sock = %sock_path,
|
||||
"Rustls fetch using mask unix socket"
|
||||
);
|
||||
return fetch_via_rustls_stream(stream, host, sni, 0).await;
|
||||
}
|
||||
Ok(Err(e)) => {
|
||||
warn!(
|
||||
sni = %sni,
|
||||
sock = %sock_path,
|
||||
error = %e,
|
||||
"Rustls unix socket connect failed, falling back to TCP"
|
||||
);
|
||||
}
|
||||
Err(_) => {
|
||||
warn!(
|
||||
sni = %sni,
|
||||
sock = %sock_path,
|
||||
"Rustls unix socket connect timed out, falling back to TCP"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(not(unix))]
|
||||
let _ = unix_sock;
|
||||
|
||||
let stream = connect_tcp_with_upstream(host, port, connect_timeout, upstream).await?;
|
||||
fetch_via_rustls_stream(stream, host, sni, proxy_protocol).await
|
||||
}
|
||||
|
||||
/// Fetch real TLS metadata for the given SNI.
|
||||
///
|
||||
/// Strategy:
|
||||
@@ -527,8 +670,20 @@ pub async fn fetch_real_tls(
|
||||
sni: &str,
|
||||
connect_timeout: Duration,
|
||||
upstream: Option<std::sync::Arc<crate::transport::UpstreamManager>>,
|
||||
proxy_protocol: u8,
|
||||
unix_sock: Option<&str>,
|
||||
) -> Result<TlsFetchResult> {
|
||||
let raw_result = match fetch_via_raw_tls(host, port, sni, connect_timeout).await {
|
||||
let raw_result = match fetch_via_raw_tls(
|
||||
host,
|
||||
port,
|
||||
sni,
|
||||
connect_timeout,
|
||||
upstream.clone(),
|
||||
proxy_protocol,
|
||||
unix_sock,
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(res) => Some(res),
|
||||
Err(e) => {
|
||||
warn!(sni = %sni, error = %e, "Raw TLS fetch failed");
|
||||
@@ -536,7 +691,17 @@ pub async fn fetch_real_tls(
|
||||
}
|
||||
};
|
||||
|
||||
match fetch_via_rustls(host, port, sni, connect_timeout, upstream).await {
|
||||
match fetch_via_rustls(
|
||||
host,
|
||||
port,
|
||||
sni,
|
||||
connect_timeout,
|
||||
upstream,
|
||||
proxy_protocol,
|
||||
unix_sock,
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(rustls_result) => {
|
||||
if let Some(mut raw) = raw_result {
|
||||
raw.cert_info = rustls_result.cert_info;
|
||||
|
||||
@@ -4,4 +4,5 @@ pub mod fetcher;
|
||||
pub mod emulator;
|
||||
|
||||
pub use cache::TlsFrontCache;
|
||||
#[allow(unused_imports)]
|
||||
pub use types::{CachedTlsData, TlsFetchResult};
|
||||
|
||||
@@ -165,11 +165,10 @@ fn process_pid16() -> u16 {
|
||||
}
|
||||
|
||||
fn process_utime() -> u32 {
|
||||
let utime = std::time::SystemTime::now()
|
||||
std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.unwrap_or_default()
|
||||
.as_secs() as u32;
|
||||
utime
|
||||
.as_secs() as u32
|
||||
}
|
||||
|
||||
pub(crate) fn cbc_encrypt_padded(
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
use std::collections::HashMap;
|
||||
use std::hash::{DefaultHasher, Hash, Hasher};
|
||||
use std::net::IpAddr;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
@@ -11,7 +12,7 @@ use crate::config::ProxyConfig;
|
||||
use crate::error::Result;
|
||||
|
||||
use super::MePool;
|
||||
use super::secret::download_proxy_secret;
|
||||
use super::secret::download_proxy_secret_with_max_len;
|
||||
use crate::crypto::SecureRandom;
|
||||
use std::time::SystemTime;
|
||||
|
||||
@@ -39,15 +40,103 @@ pub struct ProxyConfigData {
|
||||
pub default_dc: Option<i32>,
|
||||
}
|
||||
|
||||
fn parse_host_port(s: &str) -> Option<(IpAddr, u16)> {
|
||||
if let Some(bracket_end) = s.rfind(']') {
|
||||
if s.starts_with('[') && bracket_end + 1 < s.len() && s.as_bytes().get(bracket_end + 1) == Some(&b':') {
|
||||
let host = &s[1..bracket_end];
|
||||
let port_str = &s[bracket_end + 2..];
|
||||
let ip = host.parse::<IpAddr>().ok()?;
|
||||
let port = port_str.parse::<u16>().ok()?;
|
||||
return Some((ip, port));
|
||||
#[derive(Debug, Default)]
|
||||
struct StableSnapshot {
|
||||
candidate_hash: Option<u64>,
|
||||
candidate_hits: u8,
|
||||
applied_hash: Option<u64>,
|
||||
}
|
||||
|
||||
impl StableSnapshot {
|
||||
fn observe(&mut self, hash: u64) -> u8 {
|
||||
if self.candidate_hash == Some(hash) {
|
||||
self.candidate_hits = self.candidate_hits.saturating_add(1);
|
||||
} else {
|
||||
self.candidate_hash = Some(hash);
|
||||
self.candidate_hits = 1;
|
||||
}
|
||||
self.candidate_hits
|
||||
}
|
||||
|
||||
fn is_applied(&self, hash: u64) -> bool {
|
||||
self.applied_hash == Some(hash)
|
||||
}
|
||||
|
||||
fn mark_applied(&mut self, hash: u64) {
|
||||
self.applied_hash = Some(hash);
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Default)]
|
||||
struct UpdaterState {
|
||||
config_v4: StableSnapshot,
|
||||
config_v6: StableSnapshot,
|
||||
secret: StableSnapshot,
|
||||
last_map_apply_at: Option<tokio::time::Instant>,
|
||||
}
|
||||
|
||||
fn hash_proxy_config(cfg: &ProxyConfigData) -> u64 {
|
||||
let mut hasher = DefaultHasher::new();
|
||||
cfg.default_dc.hash(&mut hasher);
|
||||
|
||||
let mut by_dc: Vec<(i32, Vec<(IpAddr, u16)>)> =
|
||||
cfg.map.iter().map(|(dc, addrs)| (*dc, addrs.clone())).collect();
|
||||
by_dc.sort_by_key(|(dc, _)| *dc);
|
||||
for (dc, mut addrs) in by_dc {
|
||||
dc.hash(&mut hasher);
|
||||
addrs.sort_unstable();
|
||||
for (ip, port) in addrs {
|
||||
ip.hash(&mut hasher);
|
||||
port.hash(&mut hasher);
|
||||
}
|
||||
}
|
||||
|
||||
hasher.finish()
|
||||
}
|
||||
|
||||
fn hash_secret(secret: &[u8]) -> u64 {
|
||||
let mut hasher = DefaultHasher::new();
|
||||
secret.hash(&mut hasher);
|
||||
hasher.finish()
|
||||
}
|
||||
|
||||
fn map_apply_cooldown_ready(
|
||||
last_applied: Option<tokio::time::Instant>,
|
||||
cooldown: Duration,
|
||||
) -> bool {
|
||||
if cooldown.is_zero() {
|
||||
return true;
|
||||
}
|
||||
match last_applied {
|
||||
Some(ts) => ts.elapsed() >= cooldown,
|
||||
None => true,
|
||||
}
|
||||
}
|
||||
|
||||
fn map_apply_cooldown_remaining_secs(
|
||||
last_applied: tokio::time::Instant,
|
||||
cooldown: Duration,
|
||||
) -> u64 {
|
||||
if cooldown.is_zero() {
|
||||
return 0;
|
||||
}
|
||||
cooldown
|
||||
.checked_sub(last_applied.elapsed())
|
||||
.map(|d| d.as_secs())
|
||||
.unwrap_or(0)
|
||||
}
|
||||
|
||||
fn parse_host_port(s: &str) -> Option<(IpAddr, u16)> {
|
||||
if let Some(bracket_end) = s.rfind(']')
|
||||
&& s.starts_with('[')
|
||||
&& bracket_end + 1 < s.len()
|
||||
&& s.as_bytes().get(bracket_end + 1) == Some(&b':')
|
||||
{
|
||||
let host = &s[1..bracket_end];
|
||||
let port_str = &s[bracket_end + 2..];
|
||||
let ip = host.parse::<IpAddr>().ok()?;
|
||||
let port = port_str.parse::<u16>().ok()?;
|
||||
return Some((ip, port));
|
||||
}
|
||||
|
||||
let idx = s.rfind(':')?;
|
||||
@@ -84,20 +173,18 @@ pub async fn fetch_proxy_config(url: &str) -> Result<ProxyConfigData> {
|
||||
.map_err(|e| crate::error::ProxyError::Proxy(format!("fetch_proxy_config GET failed: {e}")))?
|
||||
;
|
||||
|
||||
if let Some(date) = resp.headers().get(reqwest::header::DATE) {
|
||||
if let Ok(date_str) = date.to_str() {
|
||||
if let Ok(server_time) = httpdate::parse_http_date(date_str) {
|
||||
if let Ok(skew) = SystemTime::now().duration_since(server_time).or_else(|e| {
|
||||
server_time.duration_since(SystemTime::now()).map_err(|_| e)
|
||||
}) {
|
||||
let skew_secs = skew.as_secs();
|
||||
if skew_secs > 60 {
|
||||
warn!(skew_secs, "Time skew >60s detected from fetch_proxy_config Date header");
|
||||
} else if skew_secs > 30 {
|
||||
warn!(skew_secs, "Time skew >30s detected from fetch_proxy_config Date header");
|
||||
}
|
||||
}
|
||||
}
|
||||
if let Some(date) = resp.headers().get(reqwest::header::DATE)
|
||||
&& let Ok(date_str) = date.to_str()
|
||||
&& let Ok(server_time) = httpdate::parse_http_date(date_str)
|
||||
&& let Ok(skew) = SystemTime::now().duration_since(server_time).or_else(|e| {
|
||||
server_time.duration_since(SystemTime::now()).map_err(|_| e)
|
||||
})
|
||||
{
|
||||
let skew_secs = skew.as_secs();
|
||||
if skew_secs > 60 {
|
||||
warn!(skew_secs, "Time skew >60s detected from fetch_proxy_config Date header");
|
||||
} else if skew_secs > 30 {
|
||||
warn!(skew_secs, "Time skew >30s detected from fetch_proxy_config Date header");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -130,41 +217,110 @@ pub async fn fetch_proxy_config(url: &str) -> Result<ProxyConfigData> {
|
||||
Ok(ProxyConfigData { map, default_dc })
|
||||
}
|
||||
|
||||
async fn run_update_cycle(pool: &Arc<MePool>, rng: &Arc<SecureRandom>, cfg: &ProxyConfig) {
|
||||
async fn run_update_cycle(
|
||||
pool: &Arc<MePool>,
|
||||
rng: &Arc<SecureRandom>,
|
||||
cfg: &ProxyConfig,
|
||||
state: &mut UpdaterState,
|
||||
) {
|
||||
pool.update_runtime_reinit_policy(
|
||||
cfg.general.hardswap,
|
||||
cfg.general.me_pool_drain_ttl_secs,
|
||||
cfg.general.effective_me_pool_force_close_secs(),
|
||||
cfg.general.me_pool_min_fresh_ratio,
|
||||
cfg.general.me_hardswap_warmup_delay_min_ms,
|
||||
cfg.general.me_hardswap_warmup_delay_max_ms,
|
||||
cfg.general.me_hardswap_warmup_extra_passes,
|
||||
cfg.general.me_hardswap_warmup_pass_backoff_base_ms,
|
||||
);
|
||||
|
||||
let required_cfg_snapshots = cfg.general.me_config_stable_snapshots.max(1);
|
||||
let required_secret_snapshots = cfg.general.proxy_secret_stable_snapshots.max(1);
|
||||
let apply_cooldown = Duration::from_secs(cfg.general.me_config_apply_cooldown_secs);
|
||||
let mut maps_changed = false;
|
||||
|
||||
// Update proxy config v4
|
||||
let mut ready_v4: Option<(ProxyConfigData, u64)> = None;
|
||||
let cfg_v4 = retry_fetch("https://core.telegram.org/getProxyConfig").await;
|
||||
if let Some(cfg_v4) = cfg_v4 {
|
||||
let changed = pool.update_proxy_maps(cfg_v4.map.clone(), None).await;
|
||||
if let Some(dc) = cfg_v4.default_dc {
|
||||
pool.default_dc
|
||||
.store(dc, std::sync::atomic::Ordering::Relaxed);
|
||||
}
|
||||
if changed {
|
||||
maps_changed = true;
|
||||
info!("ME config updated (v4)");
|
||||
let cfg_v4_hash = hash_proxy_config(&cfg_v4);
|
||||
let stable_hits = state.config_v4.observe(cfg_v4_hash);
|
||||
if stable_hits < required_cfg_snapshots {
|
||||
debug!(
|
||||
stable_hits,
|
||||
required_cfg_snapshots,
|
||||
snapshot = format_args!("0x{cfg_v4_hash:016x}"),
|
||||
"ME config v4 candidate observed"
|
||||
);
|
||||
} else if state.config_v4.is_applied(cfg_v4_hash) {
|
||||
debug!(
|
||||
snapshot = format_args!("0x{cfg_v4_hash:016x}"),
|
||||
"ME config v4 stable snapshot already applied"
|
||||
);
|
||||
} else {
|
||||
debug!("ME config v4 unchanged");
|
||||
ready_v4 = Some((cfg_v4, cfg_v4_hash));
|
||||
}
|
||||
}
|
||||
|
||||
// Update proxy config v6 (optional)
|
||||
let mut ready_v6: Option<(ProxyConfigData, u64)> = None;
|
||||
let cfg_v6 = retry_fetch("https://core.telegram.org/getProxyConfigV6").await;
|
||||
if let Some(cfg_v6) = cfg_v6 {
|
||||
let changed = pool.update_proxy_maps(HashMap::new(), Some(cfg_v6.map)).await;
|
||||
if changed {
|
||||
maps_changed = true;
|
||||
info!("ME config updated (v6)");
|
||||
let cfg_v6_hash = hash_proxy_config(&cfg_v6);
|
||||
let stable_hits = state.config_v6.observe(cfg_v6_hash);
|
||||
if stable_hits < required_cfg_snapshots {
|
||||
debug!(
|
||||
stable_hits,
|
||||
required_cfg_snapshots,
|
||||
snapshot = format_args!("0x{cfg_v6_hash:016x}"),
|
||||
"ME config v6 candidate observed"
|
||||
);
|
||||
} else if state.config_v6.is_applied(cfg_v6_hash) {
|
||||
debug!(
|
||||
snapshot = format_args!("0x{cfg_v6_hash:016x}"),
|
||||
"ME config v6 stable snapshot already applied"
|
||||
);
|
||||
} else {
|
||||
debug!("ME config v6 unchanged");
|
||||
ready_v6 = Some((cfg_v6, cfg_v6_hash));
|
||||
}
|
||||
}
|
||||
|
||||
if ready_v4.is_some() || ready_v6.is_some() {
|
||||
if map_apply_cooldown_ready(state.last_map_apply_at, apply_cooldown) {
|
||||
let update_v4 = ready_v4
|
||||
.as_ref()
|
||||
.map(|(snapshot, _)| snapshot.map.clone())
|
||||
.unwrap_or_default();
|
||||
let update_v6 = ready_v6
|
||||
.as_ref()
|
||||
.map(|(snapshot, _)| snapshot.map.clone());
|
||||
|
||||
let changed = pool.update_proxy_maps(update_v4, update_v6).await;
|
||||
|
||||
if let Some((snapshot, hash)) = ready_v4 {
|
||||
if let Some(dc) = snapshot.default_dc {
|
||||
pool.default_dc
|
||||
.store(dc, std::sync::atomic::Ordering::Relaxed);
|
||||
}
|
||||
state.config_v4.mark_applied(hash);
|
||||
}
|
||||
|
||||
if let Some((_snapshot, hash)) = ready_v6 {
|
||||
state.config_v6.mark_applied(hash);
|
||||
}
|
||||
|
||||
state.last_map_apply_at = Some(tokio::time::Instant::now());
|
||||
|
||||
if changed {
|
||||
maps_changed = true;
|
||||
info!("ME config update applied after stable-gate");
|
||||
} else {
|
||||
debug!("ME config stable-gate applied with no map delta");
|
||||
}
|
||||
} else if let Some(last) = state.last_map_apply_at {
|
||||
let wait_secs = map_apply_cooldown_remaining_secs(last, apply_cooldown);
|
||||
debug!(
|
||||
wait_secs,
|
||||
"ME config stable snapshot deferred by cooldown"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -175,14 +331,37 @@ async fn run_update_cycle(pool: &Arc<MePool>, rng: &Arc<SecureRandom>, cfg: &Pro
|
||||
|
||||
pool.reset_stun_state();
|
||||
|
||||
// Update proxy-secret
|
||||
match download_proxy_secret().await {
|
||||
Ok(secret) => {
|
||||
if pool.update_secret(secret).await {
|
||||
info!("proxy-secret updated and pool reconnect scheduled");
|
||||
if cfg.general.proxy_secret_rotate_runtime {
|
||||
match download_proxy_secret_with_max_len(cfg.general.proxy_secret_len_max).await {
|
||||
Ok(secret) => {
|
||||
let secret_hash = hash_secret(&secret);
|
||||
let stable_hits = state.secret.observe(secret_hash);
|
||||
if stable_hits < required_secret_snapshots {
|
||||
debug!(
|
||||
stable_hits,
|
||||
required_secret_snapshots,
|
||||
snapshot = format_args!("0x{secret_hash:016x}"),
|
||||
"proxy-secret candidate observed"
|
||||
);
|
||||
} else if state.secret.is_applied(secret_hash) {
|
||||
debug!(
|
||||
snapshot = format_args!("0x{secret_hash:016x}"),
|
||||
"proxy-secret stable snapshot already applied"
|
||||
);
|
||||
} else {
|
||||
let rotated = pool.update_secret(secret).await;
|
||||
state.secret.mark_applied(secret_hash);
|
||||
if rotated {
|
||||
info!("proxy-secret rotated after stable-gate");
|
||||
} else {
|
||||
debug!("proxy-secret stable snapshot confirmed as unchanged");
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(e) => warn!(error = %e, "proxy-secret update failed"),
|
||||
}
|
||||
Err(e) => warn!(error = %e, "proxy-secret update failed"),
|
||||
} else {
|
||||
debug!("proxy-secret runtime rotation disabled by config");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -191,6 +370,7 @@ pub async fn me_config_updater(
|
||||
rng: Arc<SecureRandom>,
|
||||
mut config_rx: watch::Receiver<Arc<ProxyConfig>>,
|
||||
) {
|
||||
let mut state = UpdaterState::default();
|
||||
let mut update_every_secs = config_rx
|
||||
.borrow()
|
||||
.general
|
||||
@@ -207,7 +387,7 @@ pub async fn me_config_updater(
|
||||
tokio::select! {
|
||||
_ = &mut sleep => {
|
||||
let cfg = config_rx.borrow().clone();
|
||||
run_update_cycle(&pool, &rng, cfg.as_ref()).await;
|
||||
run_update_cycle(&pool, &rng, cfg.as_ref(), &mut state).await;
|
||||
let refreshed_secs = cfg.general.effective_update_every_secs().max(1);
|
||||
if refreshed_secs != update_every_secs {
|
||||
info!(
|
||||
@@ -231,6 +411,10 @@ pub async fn me_config_updater(
|
||||
cfg.general.me_pool_drain_ttl_secs,
|
||||
cfg.general.effective_me_pool_force_close_secs(),
|
||||
cfg.general.me_pool_min_fresh_ratio,
|
||||
cfg.general.me_hardswap_warmup_delay_min_ms,
|
||||
cfg.general.me_hardswap_warmup_delay_max_ms,
|
||||
cfg.general.me_hardswap_warmup_extra_passes,
|
||||
cfg.general.me_hardswap_warmup_pass_backoff_base_ms,
|
||||
);
|
||||
let new_secs = cfg.general.effective_update_every_secs().max(1);
|
||||
if new_secs == update_every_secs {
|
||||
@@ -245,7 +429,7 @@ pub async fn me_config_updater(
|
||||
);
|
||||
update_every_secs = new_secs;
|
||||
update_every = Duration::from_secs(update_every_secs);
|
||||
run_update_cycle(&pool, &rng, cfg.as_ref()).await;
|
||||
run_update_cycle(&pool, &rng, cfg.as_ref(), &mut state).await;
|
||||
next_tick = tokio::time::Instant::now() + update_every;
|
||||
} else {
|
||||
info!(
|
||||
|
||||
@@ -14,13 +14,16 @@ use tokio::net::{TcpStream, TcpSocket};
|
||||
use tokio::time::timeout;
|
||||
use tracing::{debug, info, warn};
|
||||
|
||||
use crate::config::MeSocksKdfPolicy;
|
||||
use crate::crypto::{SecureRandom, build_middleproxy_prekey, derive_middleproxy_keys, sha256};
|
||||
use crate::error::{ProxyError, Result};
|
||||
use crate::network::IpFamily;
|
||||
use crate::network::probe::is_bogon;
|
||||
use crate::protocol::constants::{
|
||||
ME_CONNECT_TIMEOUT_SECS, ME_HANDSHAKE_TIMEOUT_SECS, RPC_CRYPTO_AES_U32,
|
||||
RPC_HANDSHAKE_ERROR_U32, rpc_crypto_flags,
|
||||
};
|
||||
use crate::transport::{UpstreamEgressInfo, UpstreamRouteKind};
|
||||
|
||||
use super::codec::{
|
||||
RpcChecksumMode, build_handshake_payload, build_nonce_payload, build_rpc_frame,
|
||||
@@ -43,12 +46,100 @@ pub(crate) struct HandshakeOutput {
|
||||
}
|
||||
|
||||
impl MePool {
|
||||
async fn resolve_dc_idx_for_endpoint(&self, addr: SocketAddr) -> Option<i16> {
|
||||
if addr.is_ipv4() {
|
||||
let map = self.proxy_map_v4.read().await;
|
||||
for (dc, addrs) in map.iter() {
|
||||
if addrs
|
||||
.iter()
|
||||
.any(|(ip, port)| SocketAddr::new(*ip, *port) == addr)
|
||||
{
|
||||
let abs_dc = dc.abs();
|
||||
if abs_dc > 0
|
||||
&& let Ok(dc_idx) = i16::try_from(abs_dc)
|
||||
{
|
||||
return Some(dc_idx);
|
||||
}
|
||||
}
|
||||
}
|
||||
} else {
|
||||
let map = self.proxy_map_v6.read().await;
|
||||
for (dc, addrs) in map.iter() {
|
||||
if addrs
|
||||
.iter()
|
||||
.any(|(ip, port)| SocketAddr::new(*ip, *port) == addr)
|
||||
{
|
||||
let abs_dc = dc.abs();
|
||||
if abs_dc > 0
|
||||
&& let Ok(dc_idx) = i16::try_from(abs_dc)
|
||||
{
|
||||
return Some(dc_idx);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
fn direct_bind_ip_for_stun(
|
||||
family: IpFamily,
|
||||
upstream_egress: Option<UpstreamEgressInfo>,
|
||||
) -> Option<IpAddr> {
|
||||
let info = upstream_egress?;
|
||||
if info.route_kind != UpstreamRouteKind::Direct {
|
||||
return None;
|
||||
}
|
||||
match (family, info.direct_bind_ip) {
|
||||
(IpFamily::V4, Some(IpAddr::V4(ip))) => Some(IpAddr::V4(ip)),
|
||||
(IpFamily::V6, Some(IpAddr::V6(ip))) => Some(IpAddr::V6(ip)),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn select_socks_bound_addr(
|
||||
family: IpFamily,
|
||||
upstream_egress: Option<UpstreamEgressInfo>,
|
||||
) -> Option<SocketAddr> {
|
||||
let info = upstream_egress?;
|
||||
if !matches!(
|
||||
info.route_kind,
|
||||
UpstreamRouteKind::Socks4 | UpstreamRouteKind::Socks5
|
||||
) {
|
||||
return None;
|
||||
}
|
||||
let bound = info.socks_bound_addr?;
|
||||
let family_matches = matches!(
|
||||
(family, bound.ip()),
|
||||
(IpFamily::V4, IpAddr::V4(_)) | (IpFamily::V6, IpAddr::V6(_))
|
||||
);
|
||||
if !family_matches || is_bogon(bound.ip()) || bound.ip().is_unspecified() {
|
||||
return None;
|
||||
}
|
||||
Some(bound)
|
||||
}
|
||||
|
||||
fn is_socks_route(upstream_egress: Option<UpstreamEgressInfo>) -> bool {
|
||||
matches!(
|
||||
upstream_egress.map(|info| info.route_kind),
|
||||
Some(UpstreamRouteKind::Socks4 | UpstreamRouteKind::Socks5)
|
||||
)
|
||||
}
|
||||
|
||||
/// TCP connect with timeout + return RTT in milliseconds.
|
||||
pub(crate) async fn connect_tcp(&self, addr: SocketAddr) -> Result<(TcpStream, f64)> {
|
||||
pub(crate) async fn connect_tcp(
|
||||
&self,
|
||||
addr: SocketAddr,
|
||||
) -> Result<(TcpStream, f64, Option<UpstreamEgressInfo>)> {
|
||||
let start = Instant::now();
|
||||
let connect_fut = async {
|
||||
if addr.is_ipv6() {
|
||||
if let Some(v6) = self.detected_ipv6 {
|
||||
let (stream, upstream_egress) = if let Some(upstream) = &self.upstream {
|
||||
let dc_idx = self.resolve_dc_idx_for_endpoint(addr).await;
|
||||
let (stream, egress) = upstream.connect_with_details(addr, dc_idx, None).await?;
|
||||
(stream, Some(egress))
|
||||
} else {
|
||||
let connect_fut = async {
|
||||
if addr.is_ipv6()
|
||||
&& let Some(v6) = self.detected_ipv6
|
||||
{
|
||||
match TcpSocket::new_v6() {
|
||||
Ok(sock) => {
|
||||
if let Err(e) = sock.bind(SocketAddr::new(IpAddr::V6(v6), 0)) {
|
||||
@@ -63,13 +154,17 @@ impl MePool {
|
||||
Err(e) => debug!(error = %e, "ME IPv6 socket creation failed, falling back to default connect"),
|
||||
}
|
||||
}
|
||||
}
|
||||
TcpStream::connect(addr).await
|
||||
TcpStream::connect(addr).await
|
||||
};
|
||||
|
||||
let stream = timeout(Duration::from_secs(ME_CONNECT_TIMEOUT_SECS), connect_fut)
|
||||
.await
|
||||
.map_err(|_| ProxyError::ConnectionTimeout {
|
||||
addr: addr.to_string(),
|
||||
})??;
|
||||
(stream, None)
|
||||
};
|
||||
|
||||
let stream = timeout(Duration::from_secs(ME_CONNECT_TIMEOUT_SECS), connect_fut)
|
||||
.await
|
||||
.map_err(|_| ProxyError::ConnectionTimeout { addr: addr.to_string() })??;
|
||||
let connect_ms = start.elapsed().as_secs_f64() * 1000.0;
|
||||
stream.set_nodelay(true).ok();
|
||||
if let Err(e) = Self::configure_keepalive(&stream) {
|
||||
@@ -79,7 +174,7 @@ impl MePool {
|
||||
if let Err(e) = Self::configure_user_timeout(stream.as_raw_fd()) {
|
||||
warn!(error = %e, "ME TCP_USER_TIMEOUT setup failed");
|
||||
}
|
||||
Ok((stream, connect_ms))
|
||||
Ok((stream, connect_ms, upstream_egress))
|
||||
}
|
||||
|
||||
fn configure_keepalive(stream: &TcpStream) -> std::io::Result<()> {
|
||||
@@ -117,12 +212,14 @@ impl MePool {
|
||||
&self,
|
||||
stream: TcpStream,
|
||||
addr: SocketAddr,
|
||||
upstream_egress: Option<UpstreamEgressInfo>,
|
||||
rng: &SecureRandom,
|
||||
) -> Result<HandshakeOutput> {
|
||||
let hs_start = Instant::now();
|
||||
|
||||
let local_addr = stream.local_addr().map_err(ProxyError::Io)?;
|
||||
let peer_addr = stream.peer_addr().map_err(ProxyError::Io)?;
|
||||
let transport_peer_addr = stream.peer_addr().map_err(ProxyError::Io)?;
|
||||
let peer_addr = addr;
|
||||
|
||||
let _ = self.maybe_detect_nat_ip(local_addr.ip()).await;
|
||||
let family = if local_addr.ip().is_ipv4() {
|
||||
@@ -130,8 +227,32 @@ impl MePool {
|
||||
} else {
|
||||
IpFamily::V6
|
||||
};
|
||||
let reflected = if self.nat_probe {
|
||||
self.maybe_reflect_public_addr(family).await
|
||||
let is_socks_route = Self::is_socks_route(upstream_egress);
|
||||
let socks_bound_addr = Self::select_socks_bound_addr(family, upstream_egress);
|
||||
let reflected = if let Some(bound) = socks_bound_addr {
|
||||
Some(bound)
|
||||
} else if is_socks_route {
|
||||
match self.socks_kdf_policy() {
|
||||
MeSocksKdfPolicy::Strict => {
|
||||
self.stats.increment_me_socks_kdf_strict_reject();
|
||||
return Err(ProxyError::InvalidHandshake(
|
||||
"SOCKS route returned no valid BND.ADDR for ME KDF (strict policy)"
|
||||
.to_string(),
|
||||
));
|
||||
}
|
||||
MeSocksKdfPolicy::Compat => {
|
||||
self.stats.increment_me_socks_kdf_compat_fallback();
|
||||
if self.nat_probe {
|
||||
let bind_ip = Self::direct_bind_ip_for_stun(family, upstream_egress);
|
||||
self.maybe_reflect_public_addr(family, bind_ip).await
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
} else if self.nat_probe {
|
||||
let bind_ip = Self::direct_bind_ip_for_stun(family, upstream_egress);
|
||||
self.maybe_reflect_public_addr(family, bind_ip).await
|
||||
} else {
|
||||
None
|
||||
};
|
||||
@@ -197,7 +318,9 @@ impl MePool {
|
||||
%local_addr_nat,
|
||||
reflected_ip = reflected.map(|r| r.ip()).as_ref().map(ToString::to_string),
|
||||
%peer_addr,
|
||||
%transport_peer_addr,
|
||||
%peer_addr_nat,
|
||||
socks_bound_addr = socks_bound_addr.map(|v| v.to_string()),
|
||||
key_selector = format_args!("0x{ks:08x}"),
|
||||
crypto_schema = format_args!("0x{schema:08x}"),
|
||||
skew_secs = skew,
|
||||
@@ -206,7 +329,11 @@ impl MePool {
|
||||
|
||||
let ts_bytes = crypto_ts.to_le_bytes();
|
||||
let server_port_bytes = peer_addr_nat.port().to_le_bytes();
|
||||
let client_port_bytes = local_addr_nat.port().to_le_bytes();
|
||||
let client_port_for_kdf = socks_bound_addr
|
||||
.map(|bound| bound.port())
|
||||
.filter(|port| *port != 0)
|
||||
.unwrap_or(local_addr_nat.port());
|
||||
let client_port_bytes = client_port_for_kdf.to_le_bytes();
|
||||
|
||||
let server_ip = extract_ip_material(peer_addr_nat);
|
||||
let client_ip = extract_ip_material(local_addr_nat);
|
||||
|
||||
@@ -1,10 +1,9 @@
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::collections::HashMap;
|
||||
use std::net::SocketAddr;
|
||||
use std::sync::Arc;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use tracing::{debug, info, warn};
|
||||
use rand::seq::SliceRandom;
|
||||
use rand::Rng;
|
||||
|
||||
use crate::crypto::SecureRandom;
|
||||
@@ -14,6 +13,7 @@ use super::MePool;
|
||||
|
||||
const HEALTH_INTERVAL_SECS: u64 = 1;
|
||||
const JITTER_FRAC_NUM: u64 = 2; // jitter up to 50% of backoff
|
||||
#[allow(dead_code)]
|
||||
const MAX_CONCURRENT_PER_DC_DEFAULT: usize = 1;
|
||||
|
||||
pub async fn me_health_monitor(pool: Arc<MePool>, rng: Arc<SecureRandom>, _min_connections: usize) {
|
||||
@@ -22,6 +22,7 @@ pub async fn me_health_monitor(pool: Arc<MePool>, rng: Arc<SecureRandom>, _min_c
|
||||
let mut inflight: HashMap<(i32, IpFamily), usize> = HashMap::new();
|
||||
loop {
|
||||
tokio::time::sleep(Duration::from_secs(HEALTH_INTERVAL_SECS)).await;
|
||||
pool.prune_closed_writers().await;
|
||||
check_family(
|
||||
IpFamily::V4,
|
||||
&pool,
|
||||
@@ -63,38 +64,50 @@ async fn check_family(
|
||||
IpFamily::V4 => pool.proxy_map_v4.read().await.clone(),
|
||||
IpFamily::V6 => pool.proxy_map_v6.read().await.clone(),
|
||||
};
|
||||
let writer_addrs: HashSet<SocketAddr> = pool
|
||||
|
||||
let mut dc_endpoints = HashMap::<i32, Vec<SocketAddr>>::new();
|
||||
for (dc, addrs) in map {
|
||||
let entry = dc_endpoints.entry(dc.abs()).or_default();
|
||||
for (ip, port) in addrs {
|
||||
entry.push(SocketAddr::new(ip, port));
|
||||
}
|
||||
}
|
||||
for endpoints in dc_endpoints.values_mut() {
|
||||
endpoints.sort_unstable();
|
||||
endpoints.dedup();
|
||||
}
|
||||
|
||||
let mut live_addr_counts = HashMap::<SocketAddr, usize>::new();
|
||||
for writer in pool
|
||||
.writers
|
||||
.read()
|
||||
.await
|
||||
.iter()
|
||||
.filter(|w| !w.draining.load(std::sync::atomic::Ordering::Relaxed))
|
||||
.map(|w| w.addr)
|
||||
.collect();
|
||||
{
|
||||
*live_addr_counts.entry(writer.addr).or_insert(0) += 1;
|
||||
}
|
||||
|
||||
let entries: Vec<(i32, Vec<SocketAddr>)> = map
|
||||
.iter()
|
||||
.map(|(dc, addrs)| {
|
||||
let list = addrs
|
||||
.iter()
|
||||
.map(|(ip, port)| SocketAddr::new(*ip, *port))
|
||||
.collect::<Vec<_>>();
|
||||
(*dc, list)
|
||||
})
|
||||
.collect();
|
||||
|
||||
for (dc, dc_addrs) in entries {
|
||||
let has_coverage = dc_addrs.iter().any(|a| writer_addrs.contains(a));
|
||||
if has_coverage {
|
||||
for (dc, endpoints) in dc_endpoints {
|
||||
if endpoints.is_empty() {
|
||||
continue;
|
||||
}
|
||||
let required = MePool::required_writers_for_dc(endpoints.len());
|
||||
let alive = endpoints
|
||||
.iter()
|
||||
.map(|addr| *live_addr_counts.get(addr).unwrap_or(&0))
|
||||
.sum::<usize>();
|
||||
if alive >= required {
|
||||
continue;
|
||||
}
|
||||
let missing = required - alive;
|
||||
|
||||
let key = (dc, family);
|
||||
let now = Instant::now();
|
||||
if let Some(ts) = next_attempt.get(&key) {
|
||||
if now < *ts {
|
||||
continue;
|
||||
}
|
||||
if let Some(ts) = next_attempt.get(&key)
|
||||
&& now < *ts
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
let max_concurrent = pool.me_reconnect_max_concurrent_per_dc.max(1) as usize;
|
||||
@@ -103,32 +116,45 @@ async fn check_family(
|
||||
}
|
||||
*inflight.entry(key).or_insert(0) += 1;
|
||||
|
||||
let mut shuffled = dc_addrs.clone();
|
||||
shuffled.shuffle(&mut rand::rng());
|
||||
let mut success = false;
|
||||
for addr in shuffled {
|
||||
let res = tokio::time::timeout(pool.me_one_timeout, pool.connect_one(addr, rng.as_ref())).await;
|
||||
let mut restored = 0usize;
|
||||
for _ in 0..missing {
|
||||
let res = tokio::time::timeout(
|
||||
pool.me_one_timeout,
|
||||
pool.connect_endpoints_round_robin(&endpoints, rng.as_ref()),
|
||||
)
|
||||
.await;
|
||||
match res {
|
||||
Ok(Ok(())) => {
|
||||
info!(%addr, dc = %dc, ?family, "ME reconnected for DC coverage");
|
||||
Ok(true) => {
|
||||
restored += 1;
|
||||
pool.stats.increment_me_reconnect_success();
|
||||
backoff.insert(key, pool.me_reconnect_backoff_base.as_millis() as u64);
|
||||
let jitter = pool.me_reconnect_backoff_base.as_millis() as u64 / JITTER_FRAC_NUM;
|
||||
let wait = pool.me_reconnect_backoff_base
|
||||
+ Duration::from_millis(rand::rng().random_range(0..=jitter.max(1)));
|
||||
next_attempt.insert(key, now + wait);
|
||||
success = true;
|
||||
break;
|
||||
}
|
||||
Ok(Err(e)) => {
|
||||
Ok(false) => {
|
||||
pool.stats.increment_me_reconnect_attempt();
|
||||
debug!(%addr, dc = %dc, error = %e, ?family, "ME reconnect failed")
|
||||
debug!(dc = %dc, ?family, "ME round-robin reconnect failed")
|
||||
}
|
||||
Err(_) => {
|
||||
pool.stats.increment_me_reconnect_attempt();
|
||||
debug!(dc = %dc, ?family, "ME reconnect timed out");
|
||||
}
|
||||
Err(_) => debug!(%addr, dc = %dc, ?family, "ME reconnect timed out"),
|
||||
}
|
||||
}
|
||||
if !success {
|
||||
pool.stats.increment_me_reconnect_attempt();
|
||||
|
||||
let now_alive = alive + restored;
|
||||
if now_alive >= required {
|
||||
info!(
|
||||
dc = %dc,
|
||||
?family,
|
||||
alive = now_alive,
|
||||
required,
|
||||
endpoint_count = endpoints.len(),
|
||||
"ME writer floor restored for DC"
|
||||
);
|
||||
backoff.insert(key, pool.me_reconnect_backoff_base.as_millis() as u64);
|
||||
let jitter = pool.me_reconnect_backoff_base.as_millis() as u64 / JITTER_FRAC_NUM;
|
||||
let wait = pool.me_reconnect_backoff_base
|
||||
+ Duration::from_millis(rand::rng().random_range(0..=jitter.max(1)));
|
||||
next_attempt.insert(key, now + wait);
|
||||
} else {
|
||||
let curr = *backoff.get(&key).unwrap_or(&(pool.me_reconnect_backoff_base.as_millis() as u64));
|
||||
let next_ms = (curr.saturating_mul(2)).min(pool.me_reconnect_backoff_cap.as_millis() as u64);
|
||||
backoff.insert(key, next_ms);
|
||||
@@ -136,7 +162,15 @@ async fn check_family(
|
||||
let wait = Duration::from_millis(next_ms)
|
||||
+ Duration::from_millis(rand::rng().random_range(0..=jitter.max(1)));
|
||||
next_attempt.insert(key, now + wait);
|
||||
warn!(dc = %dc, backoff_ms = next_ms, ?family, "DC has no ME coverage, scheduled reconnect");
|
||||
warn!(
|
||||
dc = %dc,
|
||||
?family,
|
||||
alive = now_alive,
|
||||
required,
|
||||
endpoint_count = endpoints.len(),
|
||||
backoff_ms = next_ms,
|
||||
"DC writer floor is below required level, scheduled reconnect"
|
||||
);
|
||||
}
|
||||
if let Some(v) = inflight.get_mut(&key) {
|
||||
*v = v.saturating_sub(1);
|
||||
|
||||
@@ -1,24 +1,31 @@
|
||||
//! Middle Proxy RPC transport.
|
||||
|
||||
mod codec;
|
||||
mod config_updater;
|
||||
mod handshake;
|
||||
mod health;
|
||||
mod pool;
|
||||
mod pool_config;
|
||||
mod pool_init;
|
||||
mod pool_nat;
|
||||
mod pool_refill;
|
||||
mod pool_reinit;
|
||||
mod pool_writer;
|
||||
mod ping;
|
||||
mod reader;
|
||||
mod registry;
|
||||
mod rotation;
|
||||
mod send;
|
||||
mod secret;
|
||||
mod rotation;
|
||||
mod config_updater;
|
||||
mod wire;
|
||||
|
||||
use bytes::Bytes;
|
||||
|
||||
pub use health::me_health_monitor;
|
||||
pub use ping::{run_me_ping, format_sample_line, MePingReport, MePingSample, MePingFamily};
|
||||
#[allow(unused_imports)]
|
||||
pub use ping::{run_me_ping, format_sample_line, format_me_route, MePingReport, MePingSample, MePingFamily};
|
||||
pub use pool::MePool;
|
||||
#[allow(unused_imports)]
|
||||
pub use pool_nat::{stun_probe, detect_public_ip};
|
||||
pub use registry::ConnRegistry;
|
||||
pub use secret::fetch_proxy_secret;
|
||||
|
||||
@@ -2,8 +2,12 @@ use std::collections::HashMap;
|
||||
use std::net::{IpAddr, SocketAddr};
|
||||
use std::sync::Arc;
|
||||
|
||||
use tokio::net::UdpSocket;
|
||||
|
||||
use crate::config::{UpstreamConfig, UpstreamType};
|
||||
use crate::crypto::SecureRandom;
|
||||
use crate::error::ProxyError;
|
||||
use crate::transport::{UpstreamEgressInfo, UpstreamRouteKind};
|
||||
|
||||
use super::MePool;
|
||||
|
||||
@@ -17,6 +21,7 @@ pub enum MePingFamily {
|
||||
pub struct MePingSample {
|
||||
pub dc: i32,
|
||||
pub addr: SocketAddr,
|
||||
pub route: Option<String>,
|
||||
pub connect_ms: Option<f64>,
|
||||
pub handshake_ms: Option<f64>,
|
||||
pub error: Option<String>,
|
||||
@@ -24,6 +29,7 @@ pub struct MePingSample {
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
#[allow(dead_code)]
|
||||
pub struct MePingReport {
|
||||
pub dc: i32,
|
||||
pub family: MePingFamily,
|
||||
@@ -49,6 +55,208 @@ pub fn format_sample_line(sample: &MePingSample) -> String {
|
||||
}
|
||||
}
|
||||
|
||||
fn format_direct_with_config(
|
||||
interface: &Option<String>,
|
||||
bind_addresses: &Option<Vec<String>>,
|
||||
) -> Option<String> {
|
||||
let mut direct_parts: Vec<String> = Vec::new();
|
||||
if let Some(dev) = interface.as_deref().filter(|v| !v.is_empty()) {
|
||||
direct_parts.push(format!("dev={dev}"));
|
||||
}
|
||||
if let Some(src) = bind_addresses.as_ref().filter(|v| !v.is_empty()) {
|
||||
direct_parts.push(format!("src={}", src.join(",")));
|
||||
}
|
||||
if direct_parts.is_empty() {
|
||||
None
|
||||
} else {
|
||||
Some(format!("direct {}", direct_parts.join(" ")))
|
||||
}
|
||||
}
|
||||
|
||||
fn pick_target_for_family(reports: &[MePingReport], family: MePingFamily) -> Option<SocketAddr> {
|
||||
reports.iter().find_map(|report| {
|
||||
if report.family != family {
|
||||
return None;
|
||||
}
|
||||
report
|
||||
.samples
|
||||
.iter()
|
||||
.find(|s| s.error.is_none() && s.handshake_ms.is_some())
|
||||
.map(|s| s.addr)
|
||||
})
|
||||
}
|
||||
|
||||
fn route_from_egress(egress: Option<UpstreamEgressInfo>) -> Option<String> {
|
||||
let info = egress?;
|
||||
match info.route_kind {
|
||||
UpstreamRouteKind::Direct => {
|
||||
let src_ip = info
|
||||
.direct_bind_ip
|
||||
.or_else(|| info.local_addr.map(|addr| addr.ip()));
|
||||
let ip = src_ip?;
|
||||
let mut parts = Vec::new();
|
||||
if let Some(dev) = detect_interface_for_ip(ip) {
|
||||
parts.push(format!("dev={dev}"));
|
||||
}
|
||||
parts.push(format!("src={ip}"));
|
||||
Some(format!("direct {}", parts.join(" ")))
|
||||
}
|
||||
UpstreamRouteKind::Socks4 => {
|
||||
let route = info
|
||||
.socks_proxy_addr
|
||||
.map(|addr| format!("socks4://{addr}"))
|
||||
.unwrap_or_else(|| "socks4://unknown".to_string());
|
||||
Some(match info.socks_bound_addr {
|
||||
Some(bound) => format!("{route} bnd={bound}"),
|
||||
None => route,
|
||||
})
|
||||
}
|
||||
UpstreamRouteKind::Socks5 => {
|
||||
let route = info
|
||||
.socks_proxy_addr
|
||||
.map(|addr| format!("socks5://{addr}"))
|
||||
.unwrap_or_else(|| "socks5://unknown".to_string());
|
||||
Some(match info.socks_bound_addr {
|
||||
Some(bound) => format!("{route} bnd={bound}"),
|
||||
None => route,
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
fn detect_interface_for_ip(ip: IpAddr) -> Option<String> {
|
||||
use nix::ifaddrs::getifaddrs;
|
||||
|
||||
if let Ok(addrs) = getifaddrs() {
|
||||
for iface in addrs {
|
||||
if let Some(address) = iface.address {
|
||||
if let Some(v4) = address.as_sockaddr_in() {
|
||||
if IpAddr::V4(v4.ip()) == ip {
|
||||
return Some(iface.interface_name);
|
||||
}
|
||||
} else if let Some(v6) = address.as_sockaddr_in6() {
|
||||
if IpAddr::V6(v6.ip()) == ip {
|
||||
return Some(iface.interface_name);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
#[cfg(not(unix))]
|
||||
fn detect_interface_for_ip(_ip: IpAddr) -> Option<String> {
|
||||
None
|
||||
}
|
||||
|
||||
async fn detect_direct_route_details(
|
||||
reports: &[MePingReport],
|
||||
prefer_ipv6: bool,
|
||||
v4_ok: bool,
|
||||
v6_ok: bool,
|
||||
) -> Option<String> {
|
||||
let target_addr = if prefer_ipv6 && v6_ok {
|
||||
pick_target_for_family(reports, MePingFamily::V6)
|
||||
.or_else(|| pick_target_for_family(reports, MePingFamily::V4))
|
||||
} else if v4_ok {
|
||||
pick_target_for_family(reports, MePingFamily::V4)
|
||||
.or_else(|| pick_target_for_family(reports, MePingFamily::V6))
|
||||
} else {
|
||||
pick_target_for_family(reports, MePingFamily::V6)
|
||||
.or_else(|| pick_target_for_family(reports, MePingFamily::V4))
|
||||
}?;
|
||||
|
||||
let local_ip = if target_addr.is_ipv4() {
|
||||
let sock = UdpSocket::bind("0.0.0.0:0").await.ok()?;
|
||||
sock.connect(target_addr).await.ok()?;
|
||||
sock.local_addr().ok().map(|a| a.ip())
|
||||
} else {
|
||||
let sock = UdpSocket::bind("[::]:0").await.ok()?;
|
||||
sock.connect(target_addr).await.ok()?;
|
||||
sock.local_addr().ok().map(|a| a.ip())
|
||||
};
|
||||
|
||||
let mut parts = Vec::new();
|
||||
if let Some(ip) = local_ip {
|
||||
if let Some(dev) = detect_interface_for_ip(ip) {
|
||||
parts.push(format!("dev={dev}"));
|
||||
}
|
||||
parts.push(format!("src={ip}"));
|
||||
}
|
||||
|
||||
if parts.is_empty() {
|
||||
None
|
||||
} else {
|
||||
Some(format!("direct {}", parts.join(" ")))
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn format_me_route(
|
||||
upstreams: &[UpstreamConfig],
|
||||
reports: &[MePingReport],
|
||||
prefer_ipv6: bool,
|
||||
v4_ok: bool,
|
||||
v6_ok: bool,
|
||||
) -> String {
|
||||
if let Some(route) = reports
|
||||
.iter()
|
||||
.flat_map(|report| report.samples.iter())
|
||||
.find(|sample| sample.error.is_none() && sample.handshake_ms.is_some())
|
||||
.and_then(|sample| sample.route.clone())
|
||||
{
|
||||
return route;
|
||||
}
|
||||
|
||||
let enabled_upstreams: Vec<_> = upstreams.iter().filter(|u| u.enabled).collect();
|
||||
if enabled_upstreams.is_empty() {
|
||||
return detect_direct_route_details(reports, prefer_ipv6, v4_ok, v6_ok)
|
||||
.await
|
||||
.unwrap_or_else(|| "direct".to_string());
|
||||
}
|
||||
|
||||
if enabled_upstreams.len() == 1 {
|
||||
return match &enabled_upstreams[0].upstream_type {
|
||||
UpstreamType::Direct {
|
||||
interface,
|
||||
bind_addresses,
|
||||
} => {
|
||||
if let Some(route) = format_direct_with_config(interface, bind_addresses) {
|
||||
route
|
||||
} else {
|
||||
detect_direct_route_details(reports, prefer_ipv6, v4_ok, v6_ok)
|
||||
.await
|
||||
.unwrap_or_else(|| "direct".to_string())
|
||||
}
|
||||
}
|
||||
UpstreamType::Socks4 { address, .. } => format!("socks4://{address}"),
|
||||
UpstreamType::Socks5 { address, .. } => format!("socks5://{address}"),
|
||||
};
|
||||
}
|
||||
|
||||
let has_direct = enabled_upstreams
|
||||
.iter()
|
||||
.any(|u| matches!(u.upstream_type, UpstreamType::Direct { .. }));
|
||||
let has_socks4 = enabled_upstreams
|
||||
.iter()
|
||||
.any(|u| matches!(u.upstream_type, UpstreamType::Socks4 { .. }));
|
||||
let has_socks5 = enabled_upstreams
|
||||
.iter()
|
||||
.any(|u| matches!(u.upstream_type, UpstreamType::Socks5 { .. }));
|
||||
let mut kinds = Vec::new();
|
||||
if has_direct {
|
||||
kinds.push("direct");
|
||||
}
|
||||
if has_socks4 {
|
||||
kinds.push("socks4");
|
||||
}
|
||||
if has_socks5 {
|
||||
kinds.push("socks5");
|
||||
}
|
||||
format!("mixed upstreams ({})", kinds.join(", "))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
@@ -63,6 +271,7 @@ mod tests {
|
||||
let s = sample(MePingSample {
|
||||
dc: 4,
|
||||
addr: SocketAddr::new(IpAddr::V4(Ipv4Addr::new(1, 2, 3, 4)), 8888),
|
||||
route: Some("direct src=1.2.3.4".to_string()),
|
||||
connect_ms: Some(12.3),
|
||||
handshake_ms: Some(34.7),
|
||||
error: None,
|
||||
@@ -79,6 +288,7 @@ mod tests {
|
||||
let s = sample(MePingSample {
|
||||
dc: -5,
|
||||
addr: SocketAddr::new(IpAddr::V4(Ipv4Addr::new(5, 6, 7, 8)), 80),
|
||||
route: Some("socks5".to_string()),
|
||||
connect_ms: Some(10.0),
|
||||
handshake_ms: None,
|
||||
error: Some("handshake timeout".to_string()),
|
||||
@@ -119,11 +329,13 @@ pub async fn run_me_ping(pool: &Arc<MePool>, rng: &SecureRandom) -> Vec<MePingRe
|
||||
let mut connect_ms = None;
|
||||
let mut handshake_ms = None;
|
||||
let mut error = None;
|
||||
let mut route = None;
|
||||
|
||||
match pool.connect_tcp(addr).await {
|
||||
Ok((stream, conn_rtt)) => {
|
||||
Ok((stream, conn_rtt, upstream_egress)) => {
|
||||
connect_ms = Some(conn_rtt);
|
||||
match pool.handshake_only(stream, addr, rng).await {
|
||||
route = route_from_egress(upstream_egress);
|
||||
match pool.handshake_only(stream, addr, upstream_egress, rng).await {
|
||||
Ok(hs) => {
|
||||
handshake_ms = Some(hs.handshake_ms);
|
||||
// drop halves to close
|
||||
@@ -143,6 +355,7 @@ pub async fn run_me_ping(pool: &Arc<MePool>, rng: &SecureRandom) -> Vec<MePingRe
|
||||
samples.push(MePingSample {
|
||||
dc,
|
||||
addr,
|
||||
route,
|
||||
connect_ms,
|
||||
handshake_ms,
|
||||
error,
|
||||
|
||||
@@ -1,27 +1,20 @@
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::net::{IpAddr, Ipv6Addr, SocketAddr};
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicBool, AtomicI32, AtomicU32, AtomicU64, AtomicUsize, Ordering};
|
||||
use bytes::BytesMut;
|
||||
use rand::Rng;
|
||||
use rand::seq::SliceRandom;
|
||||
use tokio::sync::{Mutex, RwLock, mpsc, Notify};
|
||||
use tokio_util::sync::CancellationToken;
|
||||
use tracing::{debug, info, warn};
|
||||
use std::sync::atomic::{AtomicBool, AtomicI32, AtomicU8, AtomicU32, AtomicU64, AtomicUsize, Ordering};
|
||||
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
|
||||
|
||||
use tokio::sync::{Mutex, Notify, RwLock, mpsc};
|
||||
use tokio_util::sync::CancellationToken;
|
||||
|
||||
use crate::config::MeSocksKdfPolicy;
|
||||
use crate::crypto::SecureRandom;
|
||||
use crate::error::{ProxyError, Result};
|
||||
use crate::network::probe::NetworkDecision;
|
||||
use crate::network::IpFamily;
|
||||
use crate::protocol::constants::*;
|
||||
use crate::network::probe::NetworkDecision;
|
||||
use crate::transport::UpstreamManager;
|
||||
|
||||
use super::ConnRegistry;
|
||||
use super::registry::BoundConn;
|
||||
use super::codec::{RpcWriter, WriterCommand};
|
||||
use super::reader::reader_loop;
|
||||
const ME_ACTIVE_PING_SECS: u64 = 25;
|
||||
const ME_ACTIVE_PING_JITTER_SECS: i64 = 5;
|
||||
use super::codec::WriterCommand;
|
||||
|
||||
#[derive(Clone)]
|
||||
pub struct MeWriter {
|
||||
@@ -36,11 +29,13 @@ pub struct MeWriter {
|
||||
pub allow_drain_fallback: Arc<AtomicBool>,
|
||||
}
|
||||
|
||||
#[allow(dead_code)]
|
||||
pub struct MePool {
|
||||
pub(super) registry: Arc<ConnRegistry>,
|
||||
pub(super) writers: Arc<RwLock<Vec<MeWriter>>>,
|
||||
pub(super) rr: AtomicU64,
|
||||
pub(super) decision: NetworkDecision,
|
||||
pub(super) upstream: Option<Arc<UpstreamManager>>,
|
||||
pub(super) rng: Arc<SecureRandom>,
|
||||
pub(super) proxy_tag: Option<Vec<u8>>,
|
||||
pub(super) proxy_secret: Arc<RwLock<Vec<u8>>>,
|
||||
@@ -49,6 +44,8 @@ pub struct MePool {
|
||||
pub(super) nat_probe: bool,
|
||||
pub(super) nat_stun: Option<String>,
|
||||
pub(super) nat_stun_servers: Vec<String>,
|
||||
pub(super) nat_stun_live_servers: Arc<RwLock<Vec<String>>>,
|
||||
pub(super) nat_probe_concurrency: usize,
|
||||
pub(super) detected_ipv6: Option<Ipv6Addr>,
|
||||
pub(super) nat_probe_attempts: std::sync::atomic::AtomicU8,
|
||||
pub(super) nat_probe_disabled: std::sync::atomic::AtomicBool,
|
||||
@@ -74,6 +71,7 @@ pub struct MePool {
|
||||
pub(super) rtt_stats: Arc<Mutex<HashMap<u64, (f64, f64)>>>,
|
||||
pub(super) nat_reflection_cache: Arc<Mutex<NatReflectionCache>>,
|
||||
pub(super) writer_available: Arc<Notify>,
|
||||
pub(super) refill_inflight: Arc<Mutex<HashSet<SocketAddr>>>,
|
||||
pub(super) conn_count: AtomicUsize,
|
||||
pub(super) stats: Arc<crate::stats::Stats>,
|
||||
pub(super) generation: AtomicU64,
|
||||
@@ -81,6 +79,11 @@ pub struct MePool {
|
||||
pub(super) me_pool_drain_ttl_secs: AtomicU64,
|
||||
pub(super) me_pool_force_close_secs: AtomicU64,
|
||||
pub(super) me_pool_min_fresh_ratio_permille: AtomicU32,
|
||||
pub(super) me_hardswap_warmup_delay_min_ms: AtomicU64,
|
||||
pub(super) me_hardswap_warmup_delay_max_ms: AtomicU64,
|
||||
pub(super) me_hardswap_warmup_extra_passes: AtomicU32,
|
||||
pub(super) me_hardswap_warmup_pass_backoff_base_ms: AtomicU64,
|
||||
pub(super) me_socks_kdf_policy: AtomicU8,
|
||||
pool_size: usize,
|
||||
}
|
||||
|
||||
@@ -96,11 +99,11 @@ impl MePool {
|
||||
(clamped * 1000.0).round() as u32
|
||||
}
|
||||
|
||||
fn permille_to_ratio(permille: u32) -> f32 {
|
||||
pub(super) fn permille_to_ratio(permille: u32) -> f32 {
|
||||
(permille.min(1000) as f32) / 1000.0
|
||||
}
|
||||
|
||||
fn now_epoch_secs() -> u64 {
|
||||
pub(super) fn now_epoch_secs() -> u64 {
|
||||
SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.unwrap_or_default()
|
||||
@@ -114,6 +117,7 @@ impl MePool {
|
||||
nat_probe: bool,
|
||||
nat_stun: Option<String>,
|
||||
nat_stun_servers: Vec<String>,
|
||||
nat_probe_concurrency: usize,
|
||||
detected_ipv6: Option<Ipv6Addr>,
|
||||
me_one_retry: u8,
|
||||
me_one_timeout_ms: u64,
|
||||
@@ -121,6 +125,7 @@ impl MePool {
|
||||
proxy_map_v6: HashMap<i32, Vec<(IpAddr, u16)>>,
|
||||
default_dc: Option<i32>,
|
||||
decision: NetworkDecision,
|
||||
upstream: Option<Arc<UpstreamManager>>,
|
||||
rng: Arc<SecureRandom>,
|
||||
stats: Arc<crate::stats::Stats>,
|
||||
me_keepalive_enabled: bool,
|
||||
@@ -138,12 +143,27 @@ impl MePool {
|
||||
me_pool_drain_ttl_secs: u64,
|
||||
me_pool_force_close_secs: u64,
|
||||
me_pool_min_fresh_ratio: f32,
|
||||
me_hardswap_warmup_delay_min_ms: u64,
|
||||
me_hardswap_warmup_delay_max_ms: u64,
|
||||
me_hardswap_warmup_extra_passes: u8,
|
||||
me_hardswap_warmup_pass_backoff_base_ms: u64,
|
||||
me_socks_kdf_policy: MeSocksKdfPolicy,
|
||||
me_route_backpressure_base_timeout_ms: u64,
|
||||
me_route_backpressure_high_timeout_ms: u64,
|
||||
me_route_backpressure_high_watermark_pct: u8,
|
||||
) -> Arc<Self> {
|
||||
let registry = Arc::new(ConnRegistry::new());
|
||||
registry.update_route_backpressure_policy(
|
||||
me_route_backpressure_base_timeout_ms,
|
||||
me_route_backpressure_high_timeout_ms,
|
||||
me_route_backpressure_high_watermark_pct,
|
||||
);
|
||||
Arc::new(Self {
|
||||
registry: Arc::new(ConnRegistry::new()),
|
||||
registry,
|
||||
writers: Arc::new(RwLock::new(Vec::new())),
|
||||
rr: AtomicU64::new(0),
|
||||
decision,
|
||||
upstream,
|
||||
rng,
|
||||
proxy_tag,
|
||||
proxy_secret: Arc::new(RwLock::new(proxy_secret)),
|
||||
@@ -152,6 +172,8 @@ impl MePool {
|
||||
nat_probe,
|
||||
nat_stun,
|
||||
nat_stun_servers,
|
||||
nat_stun_live_servers: Arc::new(RwLock::new(Vec::new())),
|
||||
nat_probe_concurrency: nat_probe_concurrency.max(1),
|
||||
detected_ipv6,
|
||||
nat_probe_attempts: std::sync::atomic::AtomicU8::new(0),
|
||||
nat_probe_disabled: std::sync::atomic::AtomicBool::new(false),
|
||||
@@ -179,12 +201,22 @@ impl MePool {
|
||||
rtt_stats: Arc::new(Mutex::new(HashMap::new())),
|
||||
nat_reflection_cache: Arc::new(Mutex::new(NatReflectionCache::default())),
|
||||
writer_available: Arc::new(Notify::new()),
|
||||
refill_inflight: Arc::new(Mutex::new(HashSet::new())),
|
||||
conn_count: AtomicUsize::new(0),
|
||||
generation: AtomicU64::new(1),
|
||||
hardswap: AtomicBool::new(hardswap),
|
||||
me_pool_drain_ttl_secs: AtomicU64::new(me_pool_drain_ttl_secs),
|
||||
me_pool_force_close_secs: AtomicU64::new(me_pool_force_close_secs),
|
||||
me_pool_min_fresh_ratio_permille: AtomicU32::new(Self::ratio_to_permille(me_pool_min_fresh_ratio)),
|
||||
me_pool_min_fresh_ratio_permille: AtomicU32::new(Self::ratio_to_permille(
|
||||
me_pool_min_fresh_ratio,
|
||||
)),
|
||||
me_hardswap_warmup_delay_min_ms: AtomicU64::new(me_hardswap_warmup_delay_min_ms),
|
||||
me_hardswap_warmup_delay_max_ms: AtomicU64::new(me_hardswap_warmup_delay_max_ms),
|
||||
me_hardswap_warmup_extra_passes: AtomicU32::new(me_hardswap_warmup_extra_passes as u32),
|
||||
me_hardswap_warmup_pass_backoff_base_ms: AtomicU64::new(
|
||||
me_hardswap_warmup_pass_backoff_base_ms,
|
||||
),
|
||||
me_socks_kdf_policy: AtomicU8::new(me_socks_kdf_policy.as_u8()),
|
||||
})
|
||||
}
|
||||
|
||||
@@ -202,18 +234,34 @@ impl MePool {
|
||||
drain_ttl_secs: u64,
|
||||
force_close_secs: u64,
|
||||
min_fresh_ratio: f32,
|
||||
hardswap_warmup_delay_min_ms: u64,
|
||||
hardswap_warmup_delay_max_ms: u64,
|
||||
hardswap_warmup_extra_passes: u8,
|
||||
hardswap_warmup_pass_backoff_base_ms: u64,
|
||||
) {
|
||||
self.hardswap.store(hardswap, Ordering::Relaxed);
|
||||
self.me_pool_drain_ttl_secs.store(drain_ttl_secs, Ordering::Relaxed);
|
||||
self.me_pool_drain_ttl_secs
|
||||
.store(drain_ttl_secs, Ordering::Relaxed);
|
||||
self.me_pool_force_close_secs
|
||||
.store(force_close_secs, Ordering::Relaxed);
|
||||
self.me_pool_min_fresh_ratio_permille
|
||||
.store(Self::ratio_to_permille(min_fresh_ratio), Ordering::Relaxed);
|
||||
self.me_hardswap_warmup_delay_min_ms
|
||||
.store(hardswap_warmup_delay_min_ms, Ordering::Relaxed);
|
||||
self.me_hardswap_warmup_delay_max_ms
|
||||
.store(hardswap_warmup_delay_max_ms, Ordering::Relaxed);
|
||||
self.me_hardswap_warmup_extra_passes
|
||||
.store(hardswap_warmup_extra_passes as u32, Ordering::Relaxed);
|
||||
self.me_hardswap_warmup_pass_backoff_base_ms
|
||||
.store(hardswap_warmup_pass_backoff_base_ms, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
pub fn reset_stun_state(&self) {
|
||||
self.nat_probe_attempts.store(0, Ordering::Relaxed);
|
||||
self.nat_probe_disabled.store(false, Ordering::Relaxed);
|
||||
if let Ok(mut live) = self.nat_stun_live_servers.try_write() {
|
||||
live.clear();
|
||||
}
|
||||
}
|
||||
|
||||
pub fn translate_our_addr(&self, addr: SocketAddr) -> SocketAddr {
|
||||
@@ -225,11 +273,31 @@ impl MePool {
|
||||
&self.registry
|
||||
}
|
||||
|
||||
fn writers_arc(&self) -> Arc<RwLock<Vec<MeWriter>>> {
|
||||
pub fn update_runtime_transport_policy(
|
||||
&self,
|
||||
socks_kdf_policy: MeSocksKdfPolicy,
|
||||
route_backpressure_base_timeout_ms: u64,
|
||||
route_backpressure_high_timeout_ms: u64,
|
||||
route_backpressure_high_watermark_pct: u8,
|
||||
) {
|
||||
self.me_socks_kdf_policy
|
||||
.store(socks_kdf_policy.as_u8(), Ordering::Relaxed);
|
||||
self.registry.update_route_backpressure_policy(
|
||||
route_backpressure_base_timeout_ms,
|
||||
route_backpressure_high_timeout_ms,
|
||||
route_backpressure_high_watermark_pct,
|
||||
);
|
||||
}
|
||||
|
||||
pub(super) fn socks_kdf_policy(&self) -> MeSocksKdfPolicy {
|
||||
MeSocksKdfPolicy::from_u8(self.me_socks_kdf_policy.load(Ordering::Relaxed))
|
||||
}
|
||||
|
||||
pub(super) fn writers_arc(&self) -> Arc<RwLock<Vec<MeWriter>>> {
|
||||
self.writers.clone()
|
||||
}
|
||||
|
||||
fn force_close_timeout(&self) -> Option<Duration> {
|
||||
pub(super) fn force_close_timeout(&self) -> Option<Duration> {
|
||||
let secs = self.me_pool_force_close_secs.load(Ordering::Relaxed);
|
||||
if secs == 0 {
|
||||
None
|
||||
@@ -238,311 +306,6 @@ impl MePool {
|
||||
}
|
||||
}
|
||||
|
||||
fn coverage_ratio(
|
||||
desired_by_dc: &HashMap<i32, HashSet<SocketAddr>>,
|
||||
active_writer_addrs: &HashSet<SocketAddr>,
|
||||
) -> (f32, Vec<i32>) {
|
||||
if desired_by_dc.is_empty() {
|
||||
return (1.0, Vec::new());
|
||||
}
|
||||
|
||||
let mut missing_dc = Vec::<i32>::new();
|
||||
let mut covered = 0usize;
|
||||
for (dc, endpoints) in desired_by_dc {
|
||||
if endpoints.is_empty() {
|
||||
continue;
|
||||
}
|
||||
if endpoints.iter().any(|addr| active_writer_addrs.contains(addr)) {
|
||||
covered += 1;
|
||||
} else {
|
||||
missing_dc.push(*dc);
|
||||
}
|
||||
}
|
||||
|
||||
missing_dc.sort_unstable();
|
||||
let total = desired_by_dc.len().max(1);
|
||||
let ratio = (covered as f32) / (total as f32);
|
||||
(ratio, missing_dc)
|
||||
}
|
||||
|
||||
pub async fn reconcile_connections(self: &Arc<Self>, rng: &SecureRandom) {
|
||||
let writers = self.writers.read().await;
|
||||
let current: HashSet<SocketAddr> = writers
|
||||
.iter()
|
||||
.filter(|w| !w.draining.load(Ordering::Relaxed))
|
||||
.map(|w| w.addr)
|
||||
.collect();
|
||||
drop(writers);
|
||||
|
||||
for family in self.family_order() {
|
||||
let map = self.proxy_map_for_family(family).await;
|
||||
for (_dc, addrs) in map.iter() {
|
||||
let dc_addrs: Vec<SocketAddr> = addrs
|
||||
.iter()
|
||||
.map(|(ip, port)| SocketAddr::new(*ip, *port))
|
||||
.collect();
|
||||
if !dc_addrs.iter().any(|a| current.contains(a)) {
|
||||
let mut shuffled = dc_addrs.clone();
|
||||
shuffled.shuffle(&mut rand::rng());
|
||||
for addr in shuffled {
|
||||
if self.connect_one(addr, rng).await.is_ok() {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if !self.decision.effective_multipath && !current.is_empty() {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn desired_dc_endpoints(&self) -> HashMap<i32, HashSet<SocketAddr>> {
|
||||
let mut out: HashMap<i32, HashSet<SocketAddr>> = HashMap::new();
|
||||
|
||||
if self.decision.ipv4_me {
|
||||
let map_v4 = self.proxy_map_v4.read().await.clone();
|
||||
for (dc, addrs) in map_v4 {
|
||||
let entry = out.entry(dc.abs()).or_default();
|
||||
for (ip, port) in addrs {
|
||||
entry.insert(SocketAddr::new(ip, port));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if self.decision.ipv6_me {
|
||||
let map_v6 = self.proxy_map_v6.read().await.clone();
|
||||
for (dc, addrs) in map_v6 {
|
||||
let entry = out.entry(dc.abs()).or_default();
|
||||
for (ip, port) in addrs {
|
||||
entry.insert(SocketAddr::new(ip, port));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
out
|
||||
}
|
||||
|
||||
async fn warmup_generation_for_all_dcs(
|
||||
self: &Arc<Self>,
|
||||
rng: &SecureRandom,
|
||||
generation: u64,
|
||||
desired_by_dc: &HashMap<i32, HashSet<SocketAddr>>,
|
||||
) {
|
||||
for endpoints in desired_by_dc.values() {
|
||||
if endpoints.is_empty() {
|
||||
continue;
|
||||
}
|
||||
|
||||
let has_fresh = {
|
||||
let ws = self.writers.read().await;
|
||||
ws.iter().any(|w| {
|
||||
!w.draining.load(Ordering::Relaxed)
|
||||
&& w.generation == generation
|
||||
&& endpoints.contains(&w.addr)
|
||||
})
|
||||
};
|
||||
|
||||
if has_fresh {
|
||||
continue;
|
||||
}
|
||||
|
||||
let mut shuffled: Vec<SocketAddr> = endpoints.iter().copied().collect();
|
||||
shuffled.shuffle(&mut rand::rng());
|
||||
for addr in shuffled {
|
||||
if self.connect_one(addr, rng).await.is_ok() {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn zero_downtime_reinit_after_map_change(
|
||||
self: &Arc<Self>,
|
||||
rng: &SecureRandom,
|
||||
) {
|
||||
let desired_by_dc = self.desired_dc_endpoints().await;
|
||||
if desired_by_dc.is_empty() {
|
||||
warn!("ME endpoint map is empty after update; skipping stale writer drain");
|
||||
return;
|
||||
}
|
||||
|
||||
let previous_generation = self.current_generation();
|
||||
let generation = self.generation.fetch_add(1, Ordering::Relaxed) + 1;
|
||||
let hardswap = self.hardswap.load(Ordering::Relaxed);
|
||||
|
||||
if hardswap {
|
||||
self.warmup_generation_for_all_dcs(rng, generation, &desired_by_dc)
|
||||
.await;
|
||||
} else {
|
||||
self.reconcile_connections(rng).await;
|
||||
}
|
||||
|
||||
let writers = self.writers.read().await;
|
||||
let active_writer_addrs: HashSet<SocketAddr> = writers
|
||||
.iter()
|
||||
.filter(|w| !w.draining.load(Ordering::Relaxed))
|
||||
.map(|w| w.addr)
|
||||
.collect();
|
||||
let min_ratio = Self::permille_to_ratio(
|
||||
self.me_pool_min_fresh_ratio_permille
|
||||
.load(Ordering::Relaxed),
|
||||
);
|
||||
let (coverage_ratio, missing_dc) = Self::coverage_ratio(&desired_by_dc, &active_writer_addrs);
|
||||
if !hardswap && coverage_ratio < min_ratio {
|
||||
warn!(
|
||||
previous_generation,
|
||||
generation,
|
||||
coverage_ratio = format_args!("{coverage_ratio:.3}"),
|
||||
min_ratio = format_args!("{min_ratio:.3}"),
|
||||
missing_dc = ?missing_dc,
|
||||
"ME reinit coverage below threshold; keeping stale writers"
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
if hardswap {
|
||||
let fresh_writer_addrs: HashSet<SocketAddr> = writers
|
||||
.iter()
|
||||
.filter(|w| !w.draining.load(Ordering::Relaxed))
|
||||
.filter(|w| w.generation == generation)
|
||||
.map(|w| w.addr)
|
||||
.collect();
|
||||
let (fresh_ratio, fresh_missing_dc) =
|
||||
Self::coverage_ratio(&desired_by_dc, &fresh_writer_addrs);
|
||||
if !fresh_missing_dc.is_empty() {
|
||||
warn!(
|
||||
previous_generation,
|
||||
generation,
|
||||
fresh_ratio = format_args!("{fresh_ratio:.3}"),
|
||||
missing_dc = ?fresh_missing_dc,
|
||||
"ME hardswap pending: fresh generation coverage incomplete"
|
||||
);
|
||||
return;
|
||||
}
|
||||
} else if !missing_dc.is_empty() {
|
||||
warn!(
|
||||
missing_dc = ?missing_dc,
|
||||
// Keep stale writers alive when fresh coverage is incomplete.
|
||||
"ME reinit coverage incomplete after map update; keeping stale writers"
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
let desired_addrs: HashSet<SocketAddr> = desired_by_dc
|
||||
.values()
|
||||
.flat_map(|set| set.iter().copied())
|
||||
.collect();
|
||||
|
||||
let stale_writer_ids: Vec<u64> = writers
|
||||
.iter()
|
||||
.filter(|w| !w.draining.load(Ordering::Relaxed))
|
||||
.filter(|w| {
|
||||
if hardswap {
|
||||
w.generation < generation
|
||||
} else {
|
||||
!desired_addrs.contains(&w.addr)
|
||||
}
|
||||
})
|
||||
.map(|w| w.id)
|
||||
.collect();
|
||||
drop(writers);
|
||||
|
||||
if stale_writer_ids.is_empty() {
|
||||
debug!("ME map update completed with no stale writers");
|
||||
return;
|
||||
}
|
||||
|
||||
let drain_timeout = self.force_close_timeout();
|
||||
let drain_timeout_secs = drain_timeout.map(|d| d.as_secs()).unwrap_or(0);
|
||||
info!(
|
||||
stale_writers = stale_writer_ids.len(),
|
||||
previous_generation,
|
||||
generation,
|
||||
hardswap,
|
||||
coverage_ratio = format_args!("{coverage_ratio:.3}"),
|
||||
min_ratio = format_args!("{min_ratio:.3}"),
|
||||
drain_timeout_secs,
|
||||
"ME map update covered; draining stale writers"
|
||||
);
|
||||
self.stats.increment_pool_swap_total();
|
||||
for writer_id in stale_writer_ids {
|
||||
self.mark_writer_draining_with_timeout(writer_id, drain_timeout, !hardswap)
|
||||
.await;
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn update_proxy_maps(
|
||||
&self,
|
||||
new_v4: HashMap<i32, Vec<(IpAddr, u16)>>,
|
||||
new_v6: Option<HashMap<i32, Vec<(IpAddr, u16)>>>,
|
||||
) -> bool {
|
||||
let mut changed = false;
|
||||
{
|
||||
let mut guard = self.proxy_map_v4.write().await;
|
||||
if !new_v4.is_empty() && *guard != new_v4 {
|
||||
*guard = new_v4;
|
||||
changed = true;
|
||||
}
|
||||
}
|
||||
if let Some(v6) = new_v6 {
|
||||
let mut guard = self.proxy_map_v6.write().await;
|
||||
if !v6.is_empty() && *guard != v6 {
|
||||
*guard = v6;
|
||||
changed = true;
|
||||
}
|
||||
}
|
||||
// Ensure negative DC entries mirror positives when absent (Telegram convention).
|
||||
{
|
||||
let mut guard = self.proxy_map_v4.write().await;
|
||||
let keys: Vec<i32> = guard.keys().cloned().collect();
|
||||
for k in keys.iter().cloned().filter(|k| *k > 0) {
|
||||
if !guard.contains_key(&-k) {
|
||||
if let Some(addrs) = guard.get(&k).cloned() {
|
||||
guard.insert(-k, addrs);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
{
|
||||
let mut guard = self.proxy_map_v6.write().await;
|
||||
let keys: Vec<i32> = guard.keys().cloned().collect();
|
||||
for k in keys.iter().cloned().filter(|k| *k > 0) {
|
||||
if !guard.contains_key(&-k) {
|
||||
if let Some(addrs) = guard.get(&k).cloned() {
|
||||
guard.insert(-k, addrs);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
changed
|
||||
}
|
||||
|
||||
pub async fn update_secret(self: &Arc<Self>, new_secret: Vec<u8>) -> bool {
|
||||
if new_secret.len() < 32 {
|
||||
warn!(len = new_secret.len(), "proxy-secret update ignored (too short)");
|
||||
return false;
|
||||
}
|
||||
let mut guard = self.proxy_secret.write().await;
|
||||
if *guard != new_secret {
|
||||
*guard = new_secret;
|
||||
drop(guard);
|
||||
self.reconnect_all().await;
|
||||
return true;
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
pub async fn reconnect_all(self: &Arc<Self>) {
|
||||
let ws = self.writers.read().await.clone();
|
||||
for w in ws {
|
||||
if let Ok(()) = self.connect_one(w.addr, self.rng.as_ref()).await {
|
||||
self.mark_writer_draining(w.id).await;
|
||||
tokio::time::sleep(Duration::from_secs(2)).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) async fn key_selector(&self) -> u32 {
|
||||
let secret = self.proxy_secret.read().await;
|
||||
if secret.len() >= 4 {
|
||||
@@ -572,437 +335,13 @@ impl MePool {
|
||||
order
|
||||
}
|
||||
|
||||
async fn proxy_map_for_family(&self, family: IpFamily) -> HashMap<i32, Vec<(IpAddr, u16)>> {
|
||||
pub(super) async fn proxy_map_for_family(
|
||||
&self,
|
||||
family: IpFamily,
|
||||
) -> HashMap<i32, Vec<(IpAddr, u16)>> {
|
||||
match family {
|
||||
IpFamily::V4 => self.proxy_map_v4.read().await.clone(),
|
||||
IpFamily::V6 => self.proxy_map_v6.read().await.clone(),
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn init(self: &Arc<Self>, pool_size: usize, rng: &Arc<SecureRandom>) -> Result<()> {
|
||||
let family_order = self.family_order();
|
||||
let ks = self.key_selector().await;
|
||||
info!(
|
||||
me_servers = self.proxy_map_v4.read().await.len(),
|
||||
pool_size,
|
||||
key_selector = format_args!("0x{ks:08x}"),
|
||||
secret_len = self.proxy_secret.read().await.len(),
|
||||
"Initializing ME pool"
|
||||
);
|
||||
|
||||
for family in family_order {
|
||||
let map = self.proxy_map_for_family(family).await;
|
||||
let dc_addrs: Vec<(i32, Vec<(IpAddr, u16)>)> = map
|
||||
.iter()
|
||||
.map(|(dc, addrs)| (*dc, addrs.clone()))
|
||||
.collect();
|
||||
|
||||
// Ensure at least one connection per DC; run DCs in parallel.
|
||||
let mut join = tokio::task::JoinSet::new();
|
||||
let mut dc_failures = 0usize;
|
||||
for (dc, addrs) in dc_addrs.iter().cloned() {
|
||||
if addrs.is_empty() {
|
||||
continue;
|
||||
}
|
||||
let pool = Arc::clone(self);
|
||||
let rng_clone = Arc::clone(rng);
|
||||
join.spawn(async move {
|
||||
pool.connect_primary_for_dc(dc, addrs, rng_clone).await
|
||||
});
|
||||
}
|
||||
while let Some(res) = join.join_next().await {
|
||||
if let Ok(false) = res {
|
||||
dc_failures += 1;
|
||||
}
|
||||
}
|
||||
if dc_failures > 2 {
|
||||
return Err(ProxyError::Proxy("Too many ME DC init failures, falling back to direct".into()));
|
||||
}
|
||||
|
||||
// Additional connections up to pool_size total (round-robin across DCs), staggered to de-phase lifecycles.
|
||||
if self.me_warmup_stagger_enabled {
|
||||
for (dc, addrs) in dc_addrs.iter() {
|
||||
for (ip, port) in addrs {
|
||||
if self.connection_count() >= pool_size {
|
||||
break;
|
||||
}
|
||||
let addr = SocketAddr::new(*ip, *port);
|
||||
let jitter = rand::rng().random_range(0..=self.me_warmup_step_jitter.as_millis() as u64);
|
||||
let delay_ms = self.me_warmup_step_delay.as_millis() as u64 + jitter;
|
||||
tokio::time::sleep(Duration::from_millis(delay_ms)).await;
|
||||
if let Err(e) = self.connect_one(addr, rng.as_ref()).await {
|
||||
debug!(%addr, dc = %dc, error = %e, "Extra ME connect failed (staggered)");
|
||||
}
|
||||
}
|
||||
}
|
||||
} else {
|
||||
for (dc, addrs) in dc_addrs.iter() {
|
||||
for (ip, port) in addrs {
|
||||
if self.connection_count() >= pool_size {
|
||||
break;
|
||||
}
|
||||
let addr = SocketAddr::new(*ip, *port);
|
||||
if let Err(e) = self.connect_one(addr, rng.as_ref()).await {
|
||||
debug!(%addr, dc = %dc, error = %e, "Extra ME connect failed");
|
||||
}
|
||||
}
|
||||
if self.connection_count() >= pool_size {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if !self.decision.effective_multipath && self.connection_count() > 0 {
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if self.writers.read().await.is_empty() {
|
||||
return Err(ProxyError::Proxy("No ME connections".into()));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(crate) async fn connect_one(self: &Arc<Self>, addr: SocketAddr, rng: &SecureRandom) -> Result<()> {
|
||||
let secret_len = self.proxy_secret.read().await.len();
|
||||
if secret_len < 32 {
|
||||
return Err(ProxyError::Proxy("proxy-secret too short for ME auth".into()));
|
||||
}
|
||||
|
||||
let (stream, _connect_ms) = self.connect_tcp(addr).await?;
|
||||
let hs = self.handshake_only(stream, addr, rng).await?;
|
||||
|
||||
let writer_id = self.next_writer_id.fetch_add(1, Ordering::Relaxed);
|
||||
let generation = self.current_generation();
|
||||
let cancel = CancellationToken::new();
|
||||
let degraded = Arc::new(AtomicBool::new(false));
|
||||
let draining = Arc::new(AtomicBool::new(false));
|
||||
let draining_started_at_epoch_secs = Arc::new(AtomicU64::new(0));
|
||||
let allow_drain_fallback = Arc::new(AtomicBool::new(false));
|
||||
let (tx, mut rx) = mpsc::channel::<WriterCommand>(4096);
|
||||
let mut rpc_writer = RpcWriter {
|
||||
writer: hs.wr,
|
||||
key: hs.write_key,
|
||||
iv: hs.write_iv,
|
||||
seq_no: 0,
|
||||
crc_mode: hs.crc_mode,
|
||||
};
|
||||
let cancel_wr = cancel.clone();
|
||||
tokio::spawn(async move {
|
||||
loop {
|
||||
tokio::select! {
|
||||
cmd = rx.recv() => {
|
||||
match cmd {
|
||||
Some(WriterCommand::Data(payload)) => {
|
||||
if rpc_writer.send(&payload).await.is_err() { break; }
|
||||
}
|
||||
Some(WriterCommand::DataAndFlush(payload)) => {
|
||||
if rpc_writer.send_and_flush(&payload).await.is_err() { break; }
|
||||
}
|
||||
Some(WriterCommand::Close) | None => break,
|
||||
}
|
||||
}
|
||||
_ = cancel_wr.cancelled() => break,
|
||||
}
|
||||
}
|
||||
});
|
||||
let writer = MeWriter {
|
||||
id: writer_id,
|
||||
addr,
|
||||
generation,
|
||||
tx: tx.clone(),
|
||||
cancel: cancel.clone(),
|
||||
degraded: degraded.clone(),
|
||||
draining: draining.clone(),
|
||||
draining_started_at_epoch_secs: draining_started_at_epoch_secs.clone(),
|
||||
allow_drain_fallback: allow_drain_fallback.clone(),
|
||||
};
|
||||
self.writers.write().await.push(writer.clone());
|
||||
self.conn_count.fetch_add(1, Ordering::Relaxed);
|
||||
self.writer_available.notify_one();
|
||||
|
||||
let reg = self.registry.clone();
|
||||
let writers_arc = self.writers_arc();
|
||||
let ping_tracker = self.ping_tracker.clone();
|
||||
let ping_tracker_reader = ping_tracker.clone();
|
||||
let rtt_stats = self.rtt_stats.clone();
|
||||
let stats_reader = self.stats.clone();
|
||||
let stats_ping = self.stats.clone();
|
||||
let pool = Arc::downgrade(self);
|
||||
let cancel_ping = cancel.clone();
|
||||
let tx_ping = tx.clone();
|
||||
let ping_tracker_ping = ping_tracker.clone();
|
||||
let cleanup_done = Arc::new(AtomicBool::new(false));
|
||||
let cleanup_for_reader = cleanup_done.clone();
|
||||
let cleanup_for_ping = cleanup_done.clone();
|
||||
let keepalive_enabled = self.me_keepalive_enabled;
|
||||
let keepalive_interval = self.me_keepalive_interval;
|
||||
let keepalive_jitter = self.me_keepalive_jitter;
|
||||
let cancel_reader_token = cancel.clone();
|
||||
let cancel_ping_token = cancel_ping.clone();
|
||||
|
||||
tokio::spawn(async move {
|
||||
let res = reader_loop(
|
||||
hs.rd,
|
||||
hs.read_key,
|
||||
hs.read_iv,
|
||||
hs.crc_mode,
|
||||
reg.clone(),
|
||||
BytesMut::new(),
|
||||
BytesMut::new(),
|
||||
tx.clone(),
|
||||
ping_tracker_reader,
|
||||
rtt_stats.clone(),
|
||||
stats_reader,
|
||||
writer_id,
|
||||
degraded.clone(),
|
||||
cancel_reader_token.clone(),
|
||||
)
|
||||
.await;
|
||||
if let Some(pool) = pool.upgrade() {
|
||||
if cleanup_for_reader
|
||||
.compare_exchange(false, true, Ordering::AcqRel, Ordering::Relaxed)
|
||||
.is_ok()
|
||||
{
|
||||
pool.remove_writer_and_close_clients(writer_id).await;
|
||||
}
|
||||
}
|
||||
if let Err(e) = res {
|
||||
warn!(error = %e, "ME reader ended");
|
||||
}
|
||||
let mut ws = writers_arc.write().await;
|
||||
ws.retain(|w| w.id != writer_id);
|
||||
info!(remaining = ws.len(), "Dead ME writer removed from pool");
|
||||
});
|
||||
|
||||
let pool_ping = Arc::downgrade(self);
|
||||
tokio::spawn(async move {
|
||||
let mut ping_id: i64 = rand::random::<i64>();
|
||||
// Per-writer jittered start to avoid phase sync.
|
||||
let startup_jitter = if keepalive_enabled {
|
||||
let jitter_cap_ms = keepalive_interval.as_millis() / 2;
|
||||
let effective_jitter_ms = keepalive_jitter.as_millis().min(jitter_cap_ms).max(1);
|
||||
Duration::from_millis(rand::rng().random_range(0..=effective_jitter_ms as u64))
|
||||
} else {
|
||||
let jitter = rand::rng()
|
||||
.random_range(-ME_ACTIVE_PING_JITTER_SECS..=ME_ACTIVE_PING_JITTER_SECS);
|
||||
let wait = (ME_ACTIVE_PING_SECS as i64 + jitter).max(5) as u64;
|
||||
Duration::from_secs(wait)
|
||||
};
|
||||
tokio::select! {
|
||||
_ = cancel_ping_token.cancelled() => return,
|
||||
_ = tokio::time::sleep(startup_jitter) => {}
|
||||
}
|
||||
loop {
|
||||
let wait = if keepalive_enabled {
|
||||
let jitter_cap_ms = keepalive_interval.as_millis() / 2;
|
||||
let effective_jitter_ms = keepalive_jitter.as_millis().min(jitter_cap_ms).max(1);
|
||||
keepalive_interval
|
||||
+ Duration::from_millis(
|
||||
rand::rng().random_range(0..=effective_jitter_ms as u64)
|
||||
)
|
||||
} else {
|
||||
let jitter = rand::rng()
|
||||
.random_range(-ME_ACTIVE_PING_JITTER_SECS..=ME_ACTIVE_PING_JITTER_SECS);
|
||||
let secs = (ME_ACTIVE_PING_SECS as i64 + jitter).max(5) as u64;
|
||||
Duration::from_secs(secs)
|
||||
};
|
||||
tokio::select! {
|
||||
_ = cancel_ping_token.cancelled() => {
|
||||
break;
|
||||
}
|
||||
_ = tokio::time::sleep(wait) => {}
|
||||
}
|
||||
let sent_id = ping_id;
|
||||
let mut p = Vec::with_capacity(12);
|
||||
p.extend_from_slice(&RPC_PING_U32.to_le_bytes());
|
||||
p.extend_from_slice(&sent_id.to_le_bytes());
|
||||
{
|
||||
let mut tracker = ping_tracker_ping.lock().await;
|
||||
let before = tracker.len();
|
||||
tracker.retain(|_, (ts, _)| ts.elapsed() < Duration::from_secs(120));
|
||||
let expired = before.saturating_sub(tracker.len());
|
||||
if expired > 0 {
|
||||
stats_ping.increment_me_keepalive_timeout_by(expired as u64);
|
||||
}
|
||||
tracker.insert(sent_id, (std::time::Instant::now(), writer_id));
|
||||
}
|
||||
ping_id = ping_id.wrapping_add(1);
|
||||
stats_ping.increment_me_keepalive_sent();
|
||||
if tx_ping.send(WriterCommand::DataAndFlush(p)).await.is_err() {
|
||||
stats_ping.increment_me_keepalive_failed();
|
||||
debug!("ME ping failed, removing dead writer");
|
||||
cancel_ping.cancel();
|
||||
if let Some(pool) = pool_ping.upgrade() {
|
||||
if cleanup_for_ping
|
||||
.compare_exchange(false, true, Ordering::AcqRel, Ordering::Relaxed)
|
||||
.is_ok()
|
||||
{
|
||||
pool.remove_writer_and_close_clients(writer_id).await;
|
||||
}
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn connect_primary_for_dc(
|
||||
self: Arc<Self>,
|
||||
dc: i32,
|
||||
mut addrs: Vec<(IpAddr, u16)>,
|
||||
rng: Arc<SecureRandom>,
|
||||
) -> bool {
|
||||
if addrs.is_empty() {
|
||||
return false;
|
||||
}
|
||||
addrs.shuffle(&mut rand::rng());
|
||||
for (ip, port) in addrs {
|
||||
let addr = SocketAddr::new(ip, port);
|
||||
match self.connect_one(addr, rng.as_ref()).await {
|
||||
Ok(()) => {
|
||||
info!(%addr, dc = %dc, "ME connected");
|
||||
return true;
|
||||
}
|
||||
Err(e) => warn!(%addr, dc = %dc, error = %e, "ME connect failed, trying next"),
|
||||
}
|
||||
}
|
||||
warn!(dc = %dc, "All ME servers for DC failed at init");
|
||||
false
|
||||
}
|
||||
|
||||
pub(crate) async fn remove_writer_and_close_clients(self: &Arc<Self>, writer_id: u64) {
|
||||
let conns = self.remove_writer_only(writer_id).await;
|
||||
for bound in conns {
|
||||
let _ = self.registry.route(bound.conn_id, super::MeResponse::Close).await;
|
||||
let _ = self.registry.unregister(bound.conn_id).await;
|
||||
}
|
||||
}
|
||||
|
||||
async fn remove_writer_only(&self, writer_id: u64) -> Vec<BoundConn> {
|
||||
let mut close_tx: Option<mpsc::Sender<WriterCommand>> = None;
|
||||
{
|
||||
let mut ws = self.writers.write().await;
|
||||
if let Some(pos) = ws.iter().position(|w| w.id == writer_id) {
|
||||
let w = ws.remove(pos);
|
||||
if w.draining.load(Ordering::Relaxed) {
|
||||
self.stats.decrement_pool_drain_active();
|
||||
}
|
||||
w.cancel.cancel();
|
||||
close_tx = Some(w.tx.clone());
|
||||
self.conn_count.fetch_sub(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
if let Some(tx) = close_tx {
|
||||
let _ = tx.send(WriterCommand::Close).await;
|
||||
}
|
||||
self.rtt_stats.lock().await.remove(&writer_id);
|
||||
self.registry.writer_lost(writer_id).await
|
||||
}
|
||||
|
||||
pub(crate) async fn mark_writer_draining_with_timeout(
|
||||
self: &Arc<Self>,
|
||||
writer_id: u64,
|
||||
timeout: Option<Duration>,
|
||||
allow_drain_fallback: bool,
|
||||
) {
|
||||
let timeout = timeout.filter(|d| !d.is_zero());
|
||||
let found = {
|
||||
let mut ws = self.writers.write().await;
|
||||
if let Some(w) = ws.iter_mut().find(|w| w.id == writer_id) {
|
||||
let already_draining = w.draining.swap(true, Ordering::Relaxed);
|
||||
w.allow_drain_fallback
|
||||
.store(allow_drain_fallback, Ordering::Relaxed);
|
||||
w.draining_started_at_epoch_secs
|
||||
.store(Self::now_epoch_secs(), Ordering::Relaxed);
|
||||
if !already_draining {
|
||||
self.stats.increment_pool_drain_active();
|
||||
}
|
||||
w.draining.store(true, Ordering::Relaxed);
|
||||
true
|
||||
} else {
|
||||
false
|
||||
}
|
||||
};
|
||||
|
||||
if !found {
|
||||
return;
|
||||
}
|
||||
|
||||
let timeout_secs = timeout.map(|d| d.as_secs()).unwrap_or(0);
|
||||
debug!(
|
||||
writer_id,
|
||||
timeout_secs,
|
||||
allow_drain_fallback,
|
||||
"ME writer marked draining"
|
||||
);
|
||||
|
||||
let pool = Arc::downgrade(self);
|
||||
tokio::spawn(async move {
|
||||
let deadline = timeout.map(|t| Instant::now() + t);
|
||||
loop {
|
||||
if let Some(p) = pool.upgrade() {
|
||||
if let Some(deadline_at) = deadline {
|
||||
if Instant::now() >= deadline_at {
|
||||
warn!(writer_id, "Drain timeout, force-closing");
|
||||
p.stats.increment_pool_force_close_total();
|
||||
let _ = p.remove_writer_and_close_clients(writer_id).await;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if p.registry.is_writer_empty(writer_id).await {
|
||||
let _ = p.remove_writer_only(writer_id).await;
|
||||
break;
|
||||
}
|
||||
tokio::time::sleep(Duration::from_secs(1)).await;
|
||||
} else {
|
||||
break;
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
pub(crate) async fn mark_writer_draining(self: &Arc<Self>, writer_id: u64) {
|
||||
self.mark_writer_draining_with_timeout(writer_id, Some(Duration::from_secs(300)), false)
|
||||
.await;
|
||||
}
|
||||
|
||||
pub(super) fn writer_accepts_new_binding(&self, writer: &MeWriter) -> bool {
|
||||
if !writer.draining.load(Ordering::Relaxed) {
|
||||
return true;
|
||||
}
|
||||
if !writer.allow_drain_fallback.load(Ordering::Relaxed) {
|
||||
return false;
|
||||
}
|
||||
|
||||
let ttl_secs = self.me_pool_drain_ttl_secs.load(Ordering::Relaxed);
|
||||
if ttl_secs == 0 {
|
||||
return true;
|
||||
}
|
||||
|
||||
let started = writer.draining_started_at_epoch_secs.load(Ordering::Relaxed);
|
||||
if started == 0 {
|
||||
return false;
|
||||
}
|
||||
|
||||
Self::now_epoch_secs().saturating_sub(started) <= ttl_secs
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
fn hex_dump(data: &[u8]) -> String {
|
||||
const MAX: usize = 64;
|
||||
let mut out = String::with_capacity(data.len() * 2 + 3);
|
||||
for (i, b) in data.iter().take(MAX).enumerate() {
|
||||
if i > 0 {
|
||||
out.push(' ');
|
||||
}
|
||||
out.push_str(&format!("{b:02x}"));
|
||||
}
|
||||
if data.len() > MAX {
|
||||
out.push_str(" …");
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
81
src/transport/middle_proxy/pool_config.rs
Normal file
81
src/transport/middle_proxy/pool_config.rs
Normal file
@@ -0,0 +1,81 @@
|
||||
use std::collections::HashMap;
|
||||
use std::net::IpAddr;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
use tracing::warn;
|
||||
|
||||
use super::pool::MePool;
|
||||
|
||||
impl MePool {
|
||||
pub async fn update_proxy_maps(
|
||||
&self,
|
||||
new_v4: HashMap<i32, Vec<(IpAddr, u16)>>,
|
||||
new_v6: Option<HashMap<i32, Vec<(IpAddr, u16)>>>,
|
||||
) -> bool {
|
||||
let mut changed = false;
|
||||
{
|
||||
let mut guard = self.proxy_map_v4.write().await;
|
||||
if !new_v4.is_empty() && *guard != new_v4 {
|
||||
*guard = new_v4;
|
||||
changed = true;
|
||||
}
|
||||
}
|
||||
if let Some(v6) = new_v6 {
|
||||
let mut guard = self.proxy_map_v6.write().await;
|
||||
if !v6.is_empty() && *guard != v6 {
|
||||
*guard = v6;
|
||||
changed = true;
|
||||
}
|
||||
}
|
||||
// Ensure negative DC entries mirror positives when absent (Telegram convention).
|
||||
{
|
||||
let mut guard = self.proxy_map_v4.write().await;
|
||||
let keys: Vec<i32> = guard.keys().cloned().collect();
|
||||
for k in keys.iter().cloned().filter(|k| *k > 0) {
|
||||
if !guard.contains_key(&-k)
|
||||
&& let Some(addrs) = guard.get(&k).cloned()
|
||||
{
|
||||
guard.insert(-k, addrs);
|
||||
}
|
||||
}
|
||||
}
|
||||
{
|
||||
let mut guard = self.proxy_map_v6.write().await;
|
||||
let keys: Vec<i32> = guard.keys().cloned().collect();
|
||||
for k in keys.iter().cloned().filter(|k| *k > 0) {
|
||||
if !guard.contains_key(&-k)
|
||||
&& let Some(addrs) = guard.get(&k).cloned()
|
||||
{
|
||||
guard.insert(-k, addrs);
|
||||
}
|
||||
}
|
||||
}
|
||||
changed
|
||||
}
|
||||
|
||||
pub async fn update_secret(self: &Arc<Self>, new_secret: Vec<u8>) -> bool {
|
||||
if new_secret.len() < 32 {
|
||||
warn!(len = new_secret.len(), "proxy-secret update ignored (too short)");
|
||||
return false;
|
||||
}
|
||||
let mut guard = self.proxy_secret.write().await;
|
||||
if *guard != new_secret {
|
||||
*guard = new_secret;
|
||||
drop(guard);
|
||||
self.reconnect_all().await;
|
||||
return true;
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
pub async fn reconnect_all(self: &Arc<Self>) {
|
||||
let ws = self.writers.read().await.clone();
|
||||
for w in ws {
|
||||
if let Ok(()) = self.connect_one(w.addr, self.rng.as_ref()).await {
|
||||
self.mark_writer_draining(w.id).await;
|
||||
tokio::time::sleep(Duration::from_secs(2)).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
201
src/transport/middle_proxy/pool_init.rs
Normal file
201
src/transport/middle_proxy/pool_init.rs
Normal file
@@ -0,0 +1,201 @@
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::net::{IpAddr, SocketAddr};
|
||||
use std::sync::Arc;
|
||||
|
||||
use rand::Rng;
|
||||
use rand::seq::SliceRandom;
|
||||
use tracing::{debug, info, warn};
|
||||
|
||||
use crate::crypto::SecureRandom;
|
||||
use crate::error::{ProxyError, Result};
|
||||
|
||||
use super::pool::MePool;
|
||||
|
||||
impl MePool {
|
||||
pub async fn init(self: &Arc<Self>, pool_size: usize, rng: &Arc<SecureRandom>) -> Result<()> {
|
||||
let family_order = self.family_order();
|
||||
let ks = self.key_selector().await;
|
||||
info!(
|
||||
me_servers = self.proxy_map_v4.read().await.len(),
|
||||
pool_size,
|
||||
key_selector = format_args!("0x{ks:08x}"),
|
||||
secret_len = self.proxy_secret.read().await.len(),
|
||||
"Initializing ME pool"
|
||||
);
|
||||
|
||||
for family in family_order {
|
||||
let map = self.proxy_map_for_family(family).await;
|
||||
let mut grouped_dc_addrs: HashMap<i32, Vec<(IpAddr, u16)>> = HashMap::new();
|
||||
for (dc, addrs) in map {
|
||||
if addrs.is_empty() {
|
||||
continue;
|
||||
}
|
||||
grouped_dc_addrs.entry(dc.abs()).or_default().extend(addrs);
|
||||
}
|
||||
let mut dc_addrs: Vec<(i32, Vec<(IpAddr, u16)>)> = grouped_dc_addrs
|
||||
.into_iter()
|
||||
.map(|(dc, mut addrs)| {
|
||||
addrs.sort_unstable();
|
||||
addrs.dedup();
|
||||
(dc, addrs)
|
||||
})
|
||||
.collect();
|
||||
dc_addrs.sort_unstable_by_key(|(dc, _)| *dc);
|
||||
|
||||
// Ensure at least one live writer per DC group; run missing DCs in parallel.
|
||||
let mut join = tokio::task::JoinSet::new();
|
||||
for (dc, addrs) in dc_addrs.iter().cloned() {
|
||||
if addrs.is_empty() {
|
||||
continue;
|
||||
}
|
||||
let endpoints: HashSet<SocketAddr> = addrs
|
||||
.iter()
|
||||
.map(|(ip, port)| SocketAddr::new(*ip, *port))
|
||||
.collect();
|
||||
if self.active_writer_count_for_endpoints(&endpoints).await > 0 {
|
||||
continue;
|
||||
}
|
||||
let pool = Arc::clone(self);
|
||||
let rng_clone = Arc::clone(rng);
|
||||
join.spawn(async move { pool.connect_primary_for_dc(dc, addrs, rng_clone).await });
|
||||
}
|
||||
while join.join_next().await.is_some() {}
|
||||
|
||||
let mut missing_dcs = Vec::new();
|
||||
for (dc, addrs) in &dc_addrs {
|
||||
let endpoints: HashSet<SocketAddr> = addrs
|
||||
.iter()
|
||||
.map(|(ip, port)| SocketAddr::new(*ip, *port))
|
||||
.collect();
|
||||
if self.active_writer_count_for_endpoints(&endpoints).await == 0 {
|
||||
missing_dcs.push(*dc);
|
||||
}
|
||||
}
|
||||
if !missing_dcs.is_empty() {
|
||||
return Err(ProxyError::Proxy(format!(
|
||||
"ME init incomplete: no live writers for DC groups {missing_dcs:?}"
|
||||
)));
|
||||
}
|
||||
|
||||
// Warm reserve writers asynchronously so startup does not block after first working pool is ready.
|
||||
let pool = Arc::clone(self);
|
||||
let rng_clone = Arc::clone(rng);
|
||||
let dc_addrs_bg = dc_addrs.clone();
|
||||
tokio::spawn(async move {
|
||||
if pool.me_warmup_stagger_enabled {
|
||||
for (dc, addrs) in &dc_addrs_bg {
|
||||
for (ip, port) in addrs {
|
||||
if pool.connection_count() >= pool_size {
|
||||
break;
|
||||
}
|
||||
let addr = SocketAddr::new(*ip, *port);
|
||||
let jitter = rand::rng()
|
||||
.random_range(0..=pool.me_warmup_step_jitter.as_millis() as u64);
|
||||
let delay_ms = pool.me_warmup_step_delay.as_millis() as u64 + jitter;
|
||||
tokio::time::sleep(std::time::Duration::from_millis(delay_ms)).await;
|
||||
if let Err(e) = pool.connect_one(addr, rng_clone.as_ref()).await {
|
||||
debug!(%addr, dc = %dc, error = %e, "Extra ME connect failed (staggered)");
|
||||
}
|
||||
}
|
||||
}
|
||||
} else {
|
||||
for (dc, addrs) in &dc_addrs_bg {
|
||||
for (ip, port) in addrs {
|
||||
if pool.connection_count() >= pool_size {
|
||||
break;
|
||||
}
|
||||
let addr = SocketAddr::new(*ip, *port);
|
||||
if let Err(e) = pool.connect_one(addr, rng_clone.as_ref()).await {
|
||||
debug!(%addr, dc = %dc, error = %e, "Extra ME connect failed");
|
||||
}
|
||||
}
|
||||
if pool.connection_count() >= pool_size {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
debug!(
|
||||
target_pool_size = pool_size,
|
||||
current_pool_size = pool.connection_count(),
|
||||
"Background ME reserve warmup finished"
|
||||
);
|
||||
});
|
||||
|
||||
if !self.decision.effective_multipath && self.connection_count() > 0 {
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if self.writers.read().await.is_empty() {
|
||||
return Err(ProxyError::Proxy("No ME connections".into()));
|
||||
}
|
||||
info!(
|
||||
active_writers = self.connection_count(),
|
||||
"ME primary pool ready; reserve warmup continues in background"
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn connect_primary_for_dc(
|
||||
self: Arc<Self>,
|
||||
dc: i32,
|
||||
mut addrs: Vec<(IpAddr, u16)>,
|
||||
rng: Arc<SecureRandom>,
|
||||
) -> bool {
|
||||
if addrs.is_empty() {
|
||||
return false;
|
||||
}
|
||||
addrs.shuffle(&mut rand::rng());
|
||||
if addrs.len() > 1 {
|
||||
let concurrency = 2usize;
|
||||
let mut join = tokio::task::JoinSet::new();
|
||||
let mut next_idx = 0usize;
|
||||
|
||||
while next_idx < addrs.len() || !join.is_empty() {
|
||||
while next_idx < addrs.len() && join.len() < concurrency {
|
||||
let (ip, port) = addrs[next_idx];
|
||||
next_idx += 1;
|
||||
let addr = SocketAddr::new(ip, port);
|
||||
let pool = Arc::clone(&self);
|
||||
let rng_clone = Arc::clone(&rng);
|
||||
join.spawn(async move {
|
||||
(addr, pool.connect_one(addr, rng_clone.as_ref()).await)
|
||||
});
|
||||
}
|
||||
|
||||
let Some(res) = join.join_next().await else {
|
||||
break;
|
||||
};
|
||||
match res {
|
||||
Ok((addr, Ok(()))) => {
|
||||
info!(%addr, dc = %dc, "ME connected");
|
||||
join.abort_all();
|
||||
while join.join_next().await.is_some() {}
|
||||
return true;
|
||||
}
|
||||
Ok((addr, Err(e))) => {
|
||||
warn!(%addr, dc = %dc, error = %e, "ME connect failed, trying next");
|
||||
}
|
||||
Err(e) => {
|
||||
warn!(dc = %dc, error = %e, "ME connect task failed");
|
||||
}
|
||||
}
|
||||
}
|
||||
warn!(dc = %dc, "All ME servers for DC failed at init");
|
||||
return false;
|
||||
}
|
||||
|
||||
for (ip, port) in addrs {
|
||||
let addr = SocketAddr::new(ip, port);
|
||||
match self.connect_one(addr, rng.as_ref()).await {
|
||||
Ok(()) => {
|
||||
info!(%addr, dc = %dc, "ME connected");
|
||||
return true;
|
||||
}
|
||||
Err(e) => warn!(%addr, dc = %dc, error = %e, "ME connect failed, trying next"),
|
||||
}
|
||||
}
|
||||
warn!(dc = %dc, "All ME servers for DC failed at init");
|
||||
false
|
||||
}
|
||||
}
|
||||
@@ -1,28 +1,139 @@
|
||||
use std::collections::HashMap;
|
||||
use std::net::{IpAddr, Ipv4Addr};
|
||||
use std::time::Duration;
|
||||
|
||||
use tracing::{info, warn, debug};
|
||||
use tokio::task::JoinSet;
|
||||
use tokio::time::timeout;
|
||||
use tracing::{debug, info, warn};
|
||||
|
||||
use crate::error::{ProxyError, Result};
|
||||
use crate::network::probe::is_bogon;
|
||||
use crate::network::stun::{stun_probe_dual, IpFamily, StunProbeResult};
|
||||
use crate::network::stun::{stun_probe_dual, stun_probe_family_with_bind, IpFamily};
|
||||
|
||||
use super::MePool;
|
||||
use std::time::Instant;
|
||||
|
||||
const STUN_BATCH_TIMEOUT: Duration = Duration::from_secs(5);
|
||||
|
||||
#[allow(dead_code)]
|
||||
pub async fn stun_probe(stun_addr: Option<String>) -> Result<crate::network::stun::DualStunResult> {
|
||||
let stun_addr = stun_addr.unwrap_or_else(|| "stun.l.google.com:19302".to_string());
|
||||
let stun_addr = stun_addr.unwrap_or_else(|| {
|
||||
crate::config::defaults::default_stun_servers()
|
||||
.into_iter()
|
||||
.next()
|
||||
.unwrap_or_default()
|
||||
});
|
||||
if stun_addr.is_empty() {
|
||||
return Err(ProxyError::Proxy("STUN server is not configured".to_string()));
|
||||
}
|
||||
stun_probe_dual(&stun_addr).await
|
||||
}
|
||||
|
||||
#[allow(dead_code)]
|
||||
pub async fn detect_public_ip() -> Option<IpAddr> {
|
||||
fetch_public_ipv4_with_retry().await.ok().flatten().map(IpAddr::V4)
|
||||
}
|
||||
|
||||
impl MePool {
|
||||
fn configured_stun_servers(&self) -> Vec<String> {
|
||||
if !self.nat_stun_servers.is_empty() {
|
||||
return self.nat_stun_servers.clone();
|
||||
}
|
||||
if let Some(s) = &self.nat_stun
|
||||
&& !s.trim().is_empty()
|
||||
{
|
||||
return vec![s.clone()];
|
||||
}
|
||||
Vec::new()
|
||||
}
|
||||
|
||||
async fn probe_stun_batch_for_family(
|
||||
&self,
|
||||
servers: &[String],
|
||||
family: IpFamily,
|
||||
attempt: u8,
|
||||
bind_ip: Option<IpAddr>,
|
||||
) -> (Vec<String>, Option<std::net::SocketAddr>) {
|
||||
let mut join_set = JoinSet::new();
|
||||
let mut next_idx = 0usize;
|
||||
let mut live_servers = Vec::new();
|
||||
let mut best_by_ip: HashMap<IpAddr, (usize, std::net::SocketAddr)> = HashMap::new();
|
||||
let concurrency = self.nat_probe_concurrency.max(1);
|
||||
|
||||
while next_idx < servers.len() || !join_set.is_empty() {
|
||||
while next_idx < servers.len() && join_set.len() < concurrency {
|
||||
let stun_addr = servers[next_idx].clone();
|
||||
next_idx += 1;
|
||||
join_set.spawn(async move {
|
||||
let res = timeout(
|
||||
STUN_BATCH_TIMEOUT,
|
||||
stun_probe_family_with_bind(&stun_addr, family, bind_ip),
|
||||
)
|
||||
.await;
|
||||
(stun_addr, res)
|
||||
});
|
||||
}
|
||||
|
||||
let Some(task) = join_set.join_next().await else {
|
||||
break;
|
||||
};
|
||||
|
||||
match task {
|
||||
Ok((stun_addr, Ok(Ok(picked)))) => {
|
||||
if let Some(result) = picked {
|
||||
live_servers.push(stun_addr.clone());
|
||||
let entry = best_by_ip
|
||||
.entry(result.reflected_addr.ip())
|
||||
.or_insert((0, result.reflected_addr));
|
||||
entry.0 += 1;
|
||||
debug!(
|
||||
local = %result.local_addr,
|
||||
reflected = %result.reflected_addr,
|
||||
family = ?family,
|
||||
stun = %stun_addr,
|
||||
"NAT probe: reflected address"
|
||||
);
|
||||
}
|
||||
}
|
||||
Ok((stun_addr, Ok(Err(e)))) => {
|
||||
debug!(
|
||||
error = %e,
|
||||
stun = %stun_addr,
|
||||
attempt = attempt + 1,
|
||||
"NAT probe failed, trying next server"
|
||||
);
|
||||
}
|
||||
Ok((stun_addr, Err(_))) => {
|
||||
debug!(
|
||||
stun = %stun_addr,
|
||||
attempt = attempt + 1,
|
||||
"NAT probe timeout, trying next server"
|
||||
);
|
||||
}
|
||||
Err(e) => {
|
||||
debug!(
|
||||
error = %e,
|
||||
attempt = attempt + 1,
|
||||
"NAT probe task join failed"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
live_servers.sort_unstable();
|
||||
live_servers.dedup();
|
||||
let best_reflected = best_by_ip
|
||||
.into_values()
|
||||
.max_by_key(|(count, _)| *count)
|
||||
.map(|(_, addr)| addr);
|
||||
|
||||
(live_servers, best_reflected)
|
||||
}
|
||||
|
||||
pub(super) fn translate_ip_for_nat(&self, ip: IpAddr) -> IpAddr {
|
||||
let nat_ip = self
|
||||
.nat_ip_cfg
|
||||
.or_else(|| self.nat_ip_detected.try_read().ok().and_then(|g| (*g).clone()));
|
||||
.or_else(|| self.nat_ip_detected.try_read().ok().and_then(|g| *g));
|
||||
|
||||
let Some(nat_ip) = nat_ip else {
|
||||
return ip;
|
||||
@@ -72,7 +183,7 @@ impl MePool {
|
||||
return None;
|
||||
}
|
||||
|
||||
if let Some(ip) = self.nat_ip_detected.read().await.clone() {
|
||||
if let Some(ip) = *self.nat_ip_detected.read().await {
|
||||
return Some(ip);
|
||||
}
|
||||
|
||||
@@ -96,70 +207,105 @@ impl MePool {
|
||||
pub(super) async fn maybe_reflect_public_addr(
|
||||
&self,
|
||||
family: IpFamily,
|
||||
bind_ip: Option<IpAddr>,
|
||||
) -> Option<std::net::SocketAddr> {
|
||||
const STUN_CACHE_TTL: Duration = Duration::from_secs(600);
|
||||
// Backoff window
|
||||
if let Some(until) = *self.stun_backoff_until.read().await {
|
||||
if Instant::now() < until {
|
||||
if let Ok(cache) = self.nat_reflection_cache.try_lock() {
|
||||
let slot = match family {
|
||||
IpFamily::V4 => cache.v4,
|
||||
IpFamily::V6 => cache.v6,
|
||||
};
|
||||
return slot.map(|(_, addr)| addr);
|
||||
}
|
||||
return None;
|
||||
let use_shared_cache = bind_ip.is_none();
|
||||
if !use_shared_cache {
|
||||
match (family, bind_ip) {
|
||||
(IpFamily::V4, Some(IpAddr::V4(_)))
|
||||
| (IpFamily::V6, Some(IpAddr::V6(_)))
|
||||
| (_, None) => {}
|
||||
_ => return None,
|
||||
}
|
||||
}
|
||||
// Backoff window
|
||||
if use_shared_cache
|
||||
&& let Some(until) = *self.stun_backoff_until.read().await
|
||||
&& Instant::now() < until
|
||||
{
|
||||
if let Ok(cache) = self.nat_reflection_cache.try_lock() {
|
||||
let slot = match family {
|
||||
IpFamily::V4 => cache.v4,
|
||||
IpFamily::V6 => cache.v6,
|
||||
};
|
||||
return slot.map(|(_, addr)| addr);
|
||||
}
|
||||
return None;
|
||||
}
|
||||
|
||||
if let Ok(mut cache) = self.nat_reflection_cache.try_lock() {
|
||||
if use_shared_cache
|
||||
&& let Ok(mut cache) = self.nat_reflection_cache.try_lock()
|
||||
{
|
||||
let slot = match family {
|
||||
IpFamily::V4 => &mut cache.v4,
|
||||
IpFamily::V6 => &mut cache.v6,
|
||||
};
|
||||
if let Some((ts, addr)) = slot {
|
||||
if ts.elapsed() < STUN_CACHE_TTL {
|
||||
return Some(*addr);
|
||||
}
|
||||
if let Some((ts, addr)) = slot
|
||||
&& ts.elapsed() < STUN_CACHE_TTL
|
||||
{
|
||||
return Some(*addr);
|
||||
}
|
||||
}
|
||||
|
||||
let attempt = self.nat_probe_attempts.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
|
||||
let servers = if !self.nat_stun_servers.is_empty() {
|
||||
self.nat_stun_servers.clone()
|
||||
} else if let Some(s) = &self.nat_stun {
|
||||
vec![s.clone()]
|
||||
let attempt = if use_shared_cache {
|
||||
self.nat_probe_attempts.fetch_add(1, std::sync::atomic::Ordering::Relaxed)
|
||||
} else {
|
||||
vec!["stun.l.google.com:19302".to_string()]
|
||||
0
|
||||
};
|
||||
let configured_servers = self.configured_stun_servers();
|
||||
let live_snapshot = self.nat_stun_live_servers.read().await.clone();
|
||||
let primary_servers = if live_snapshot.is_empty() {
|
||||
configured_servers.clone()
|
||||
} else {
|
||||
live_snapshot
|
||||
};
|
||||
|
||||
for stun_addr in servers {
|
||||
match stun_probe_dual(&stun_addr).await {
|
||||
Ok(res) => {
|
||||
let picked: Option<StunProbeResult> = match family {
|
||||
IpFamily::V4 => res.v4,
|
||||
IpFamily::V6 => res.v6,
|
||||
};
|
||||
if let Some(result) = picked {
|
||||
info!(local = %result.local_addr, reflected = %result.reflected_addr, family = ?family, stun = %stun_addr, "NAT probe: reflected address");
|
||||
self.nat_probe_attempts.store(0, std::sync::atomic::Ordering::Relaxed);
|
||||
if let Ok(mut cache) = self.nat_reflection_cache.try_lock() {
|
||||
let slot = match family {
|
||||
IpFamily::V4 => &mut cache.v4,
|
||||
IpFamily::V6 => &mut cache.v6,
|
||||
};
|
||||
*slot = Some((Instant::now(), result.reflected_addr));
|
||||
}
|
||||
return Some(result.reflected_addr);
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
warn!(error = %e, stun = %stun_addr, attempt = attempt + 1, "NAT probe failed, trying next server");
|
||||
}
|
||||
}
|
||||
let (mut live_servers, mut selected_reflected) = self
|
||||
.probe_stun_batch_for_family(&primary_servers, family, attempt, bind_ip)
|
||||
.await;
|
||||
|
||||
if selected_reflected.is_none() && !configured_servers.is_empty() && primary_servers != configured_servers {
|
||||
let (rediscovered_live, rediscovered_reflected) = self
|
||||
.probe_stun_batch_for_family(&configured_servers, family, attempt, bind_ip)
|
||||
.await;
|
||||
live_servers = rediscovered_live;
|
||||
selected_reflected = rediscovered_reflected;
|
||||
}
|
||||
|
||||
let live_server_count = live_servers.len();
|
||||
if !live_servers.is_empty() {
|
||||
*self.nat_stun_live_servers.write().await = live_servers;
|
||||
} else {
|
||||
self.nat_stun_live_servers.write().await.clear();
|
||||
}
|
||||
|
||||
if let Some(reflected_addr) = selected_reflected {
|
||||
if use_shared_cache {
|
||||
self.nat_probe_attempts.store(0, std::sync::atomic::Ordering::Relaxed);
|
||||
}
|
||||
info!(
|
||||
family = ?family,
|
||||
live_servers = live_server_count,
|
||||
"STUN-Quorum reached, IP: {}",
|
||||
reflected_addr.ip()
|
||||
);
|
||||
if use_shared_cache
|
||||
&& let Ok(mut cache) = self.nat_reflection_cache.try_lock()
|
||||
{
|
||||
let slot = match family {
|
||||
IpFamily::V4 => &mut cache.v4,
|
||||
IpFamily::V6 => &mut cache.v6,
|
||||
};
|
||||
*slot = Some((Instant::now(), reflected_addr));
|
||||
}
|
||||
return Some(reflected_addr);
|
||||
}
|
||||
|
||||
if use_shared_cache {
|
||||
let backoff = Duration::from_secs(60 * 2u64.pow((attempt as u32).min(6)));
|
||||
*self.stun_backoff_until.write().await = Some(Instant::now() + backoff);
|
||||
}
|
||||
let backoff = Duration::from_secs(60 * 2u64.pow((attempt as u32).min(6)));
|
||||
*self.stun_backoff_until.write().await = Some(Instant::now() + backoff);
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
159
src/transport/middle_proxy/pool_refill.rs
Normal file
159
src/transport/middle_proxy/pool_refill.rs
Normal file
@@ -0,0 +1,159 @@
|
||||
use std::collections::HashSet;
|
||||
use std::net::SocketAddr;
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::Ordering;
|
||||
|
||||
use tracing::{debug, info, warn};
|
||||
|
||||
use crate::crypto::SecureRandom;
|
||||
|
||||
use super::pool::MePool;
|
||||
|
||||
impl MePool {
|
||||
pub(super) async fn connect_endpoints_round_robin(
|
||||
self: &Arc<Self>,
|
||||
endpoints: &[SocketAddr],
|
||||
rng: &SecureRandom,
|
||||
) -> bool {
|
||||
if endpoints.is_empty() {
|
||||
return false;
|
||||
}
|
||||
let start = (self.rr.fetch_add(1, Ordering::Relaxed) as usize) % endpoints.len();
|
||||
for offset in 0..endpoints.len() {
|
||||
let idx = (start + offset) % endpoints.len();
|
||||
let addr = endpoints[idx];
|
||||
match self.connect_one(addr, rng).await {
|
||||
Ok(()) => return true,
|
||||
Err(e) => debug!(%addr, error = %e, "ME connect failed during round-robin warmup"),
|
||||
}
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
async fn endpoints_for_same_dc(&self, addr: SocketAddr) -> Vec<SocketAddr> {
|
||||
let mut target_dc = HashSet::<i32>::new();
|
||||
let mut endpoints = HashSet::<SocketAddr>::new();
|
||||
|
||||
if self.decision.ipv4_me {
|
||||
let map = self.proxy_map_v4.read().await.clone();
|
||||
for (dc, addrs) in &map {
|
||||
if addrs
|
||||
.iter()
|
||||
.any(|(ip, port)| SocketAddr::new(*ip, *port) == addr)
|
||||
{
|
||||
target_dc.insert(dc.abs());
|
||||
}
|
||||
}
|
||||
for dc in &target_dc {
|
||||
for key in [*dc, -*dc] {
|
||||
if let Some(addrs) = map.get(&key) {
|
||||
for (ip, port) in addrs {
|
||||
endpoints.insert(SocketAddr::new(*ip, *port));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if self.decision.ipv6_me {
|
||||
let map = self.proxy_map_v6.read().await.clone();
|
||||
for (dc, addrs) in &map {
|
||||
if addrs
|
||||
.iter()
|
||||
.any(|(ip, port)| SocketAddr::new(*ip, *port) == addr)
|
||||
{
|
||||
target_dc.insert(dc.abs());
|
||||
}
|
||||
}
|
||||
for dc in &target_dc {
|
||||
for key in [*dc, -*dc] {
|
||||
if let Some(addrs) = map.get(&key) {
|
||||
for (ip, port) in addrs {
|
||||
endpoints.insert(SocketAddr::new(*ip, *port));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let mut sorted: Vec<SocketAddr> = endpoints.into_iter().collect();
|
||||
sorted.sort_unstable();
|
||||
sorted
|
||||
}
|
||||
|
||||
async fn refill_writer_after_loss(self: &Arc<Self>, addr: SocketAddr) -> bool {
|
||||
let fast_retries = self.me_reconnect_fast_retry_count.max(1);
|
||||
|
||||
for attempt in 0..fast_retries {
|
||||
self.stats.increment_me_reconnect_attempt();
|
||||
match self.connect_one(addr, self.rng.as_ref()).await {
|
||||
Ok(()) => {
|
||||
self.stats.increment_me_reconnect_success();
|
||||
self.stats.increment_me_writer_restored_same_endpoint_total();
|
||||
info!(
|
||||
%addr,
|
||||
attempt = attempt + 1,
|
||||
"ME writer restored on the same endpoint"
|
||||
);
|
||||
return true;
|
||||
}
|
||||
Err(e) => {
|
||||
debug!(
|
||||
%addr,
|
||||
attempt = attempt + 1,
|
||||
error = %e,
|
||||
"ME immediate same-endpoint reconnect failed"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let dc_endpoints = self.endpoints_for_same_dc(addr).await;
|
||||
if dc_endpoints.is_empty() {
|
||||
self.stats.increment_me_refill_failed_total();
|
||||
return false;
|
||||
}
|
||||
|
||||
for attempt in 0..fast_retries {
|
||||
self.stats.increment_me_reconnect_attempt();
|
||||
if self
|
||||
.connect_endpoints_round_robin(&dc_endpoints, self.rng.as_ref())
|
||||
.await
|
||||
{
|
||||
self.stats.increment_me_reconnect_success();
|
||||
self.stats.increment_me_writer_restored_fallback_total();
|
||||
info!(
|
||||
%addr,
|
||||
attempt = attempt + 1,
|
||||
"ME writer restored via DC fallback endpoint"
|
||||
);
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
self.stats.increment_me_refill_failed_total();
|
||||
false
|
||||
}
|
||||
|
||||
pub(crate) fn trigger_immediate_refill(self: &Arc<Self>, addr: SocketAddr) {
|
||||
let pool = Arc::clone(self);
|
||||
tokio::spawn(async move {
|
||||
{
|
||||
let mut guard = pool.refill_inflight.lock().await;
|
||||
if !guard.insert(addr) {
|
||||
pool.stats.increment_me_refill_skipped_inflight_total();
|
||||
return;
|
||||
}
|
||||
}
|
||||
pool.stats.increment_me_refill_triggered_total();
|
||||
|
||||
let restored = pool.refill_writer_after_loss(addr).await;
|
||||
if !restored {
|
||||
warn!(%addr, "ME immediate refill failed");
|
||||
}
|
||||
|
||||
let mut guard = pool.refill_inflight.lock().await;
|
||||
guard.remove(&addr);
|
||||
});
|
||||
}
|
||||
}
|
||||
383
src/transport/middle_proxy/pool_reinit.rs
Normal file
383
src/transport/middle_proxy/pool_reinit.rs
Normal file
@@ -0,0 +1,383 @@
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::net::SocketAddr;
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::Ordering;
|
||||
use std::time::Duration;
|
||||
|
||||
use rand::Rng;
|
||||
use rand::seq::SliceRandom;
|
||||
use tracing::{debug, info, warn};
|
||||
|
||||
use crate::crypto::SecureRandom;
|
||||
|
||||
use super::pool::MePool;
|
||||
|
||||
impl MePool {
|
||||
fn coverage_ratio(
|
||||
desired_by_dc: &HashMap<i32, HashSet<SocketAddr>>,
|
||||
active_writer_addrs: &HashSet<SocketAddr>,
|
||||
) -> (f32, Vec<i32>) {
|
||||
if desired_by_dc.is_empty() {
|
||||
return (1.0, Vec::new());
|
||||
}
|
||||
|
||||
let mut missing_dc = Vec::<i32>::new();
|
||||
let mut covered = 0usize;
|
||||
for (dc, endpoints) in desired_by_dc {
|
||||
if endpoints.is_empty() {
|
||||
continue;
|
||||
}
|
||||
if endpoints
|
||||
.iter()
|
||||
.any(|addr| active_writer_addrs.contains(addr))
|
||||
{
|
||||
covered += 1;
|
||||
} else {
|
||||
missing_dc.push(*dc);
|
||||
}
|
||||
}
|
||||
|
||||
missing_dc.sort_unstable();
|
||||
let total = desired_by_dc.len().max(1);
|
||||
let ratio = (covered as f32) / (total as f32);
|
||||
(ratio, missing_dc)
|
||||
}
|
||||
|
||||
pub async fn reconcile_connections(self: &Arc<Self>, rng: &SecureRandom) {
|
||||
let writers = self.writers.read().await;
|
||||
let current: HashSet<SocketAddr> = writers
|
||||
.iter()
|
||||
.filter(|w| !w.draining.load(Ordering::Relaxed))
|
||||
.map(|w| w.addr)
|
||||
.collect();
|
||||
drop(writers);
|
||||
|
||||
for family in self.family_order() {
|
||||
let map = self.proxy_map_for_family(family).await;
|
||||
for (_dc, addrs) in &map {
|
||||
let dc_addrs: Vec<SocketAddr> = addrs
|
||||
.iter()
|
||||
.map(|(ip, port)| SocketAddr::new(*ip, *port))
|
||||
.collect();
|
||||
if !dc_addrs.iter().any(|a| current.contains(a)) {
|
||||
let mut shuffled = dc_addrs.clone();
|
||||
shuffled.shuffle(&mut rand::rng());
|
||||
for addr in shuffled {
|
||||
if self.connect_one(addr, rng).await.is_ok() {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if !self.decision.effective_multipath && !current.is_empty() {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn desired_dc_endpoints(&self) -> HashMap<i32, HashSet<SocketAddr>> {
|
||||
let mut out: HashMap<i32, HashSet<SocketAddr>> = HashMap::new();
|
||||
|
||||
if self.decision.ipv4_me {
|
||||
let map_v4 = self.proxy_map_v4.read().await.clone();
|
||||
for (dc, addrs) in map_v4 {
|
||||
let entry = out.entry(dc.abs()).or_default();
|
||||
for (ip, port) in addrs {
|
||||
entry.insert(SocketAddr::new(ip, port));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if self.decision.ipv6_me {
|
||||
let map_v6 = self.proxy_map_v6.read().await.clone();
|
||||
for (dc, addrs) in map_v6 {
|
||||
let entry = out.entry(dc.abs()).or_default();
|
||||
for (ip, port) in addrs {
|
||||
entry.insert(SocketAddr::new(ip, port));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
out
|
||||
}
|
||||
|
||||
pub(super) fn required_writers_for_dc(endpoint_count: usize) -> usize {
|
||||
endpoint_count.max(3)
|
||||
}
|
||||
|
||||
fn hardswap_warmup_connect_delay_ms(&self) -> u64 {
|
||||
let min_ms = self.me_hardswap_warmup_delay_min_ms.load(Ordering::Relaxed);
|
||||
let max_ms = self.me_hardswap_warmup_delay_max_ms.load(Ordering::Relaxed);
|
||||
let (min_ms, max_ms) = if min_ms <= max_ms {
|
||||
(min_ms, max_ms)
|
||||
} else {
|
||||
(max_ms, min_ms)
|
||||
};
|
||||
if min_ms == max_ms {
|
||||
return min_ms;
|
||||
}
|
||||
rand::rng().random_range(min_ms..=max_ms)
|
||||
}
|
||||
|
||||
fn hardswap_warmup_backoff_ms(&self, pass_idx: usize) -> u64 {
|
||||
let base_ms = self
|
||||
.me_hardswap_warmup_pass_backoff_base_ms
|
||||
.load(Ordering::Relaxed);
|
||||
let cap_ms = (self.me_reconnect_backoff_cap.as_millis() as u64).max(base_ms);
|
||||
let shift = (pass_idx as u32).min(20);
|
||||
let scaled = base_ms.saturating_mul(1u64 << shift);
|
||||
let core = scaled.min(cap_ms);
|
||||
let jitter = (core / 2).max(1);
|
||||
core.saturating_add(rand::rng().random_range(0..=jitter))
|
||||
}
|
||||
|
||||
async fn fresh_writer_count_for_endpoints(
|
||||
&self,
|
||||
generation: u64,
|
||||
endpoints: &HashSet<SocketAddr>,
|
||||
) -> usize {
|
||||
let ws = self.writers.read().await;
|
||||
ws.iter()
|
||||
.filter(|w| !w.draining.load(Ordering::Relaxed))
|
||||
.filter(|w| w.generation == generation)
|
||||
.filter(|w| endpoints.contains(&w.addr))
|
||||
.count()
|
||||
}
|
||||
|
||||
pub(super) async fn active_writer_count_for_endpoints(
|
||||
&self,
|
||||
endpoints: &HashSet<SocketAddr>,
|
||||
) -> usize {
|
||||
let ws = self.writers.read().await;
|
||||
ws.iter()
|
||||
.filter(|w| !w.draining.load(Ordering::Relaxed))
|
||||
.filter(|w| endpoints.contains(&w.addr))
|
||||
.count()
|
||||
}
|
||||
|
||||
async fn warmup_generation_for_all_dcs(
|
||||
self: &Arc<Self>,
|
||||
rng: &SecureRandom,
|
||||
generation: u64,
|
||||
desired_by_dc: &HashMap<i32, HashSet<SocketAddr>>,
|
||||
) {
|
||||
let extra_passes = self
|
||||
.me_hardswap_warmup_extra_passes
|
||||
.load(Ordering::Relaxed)
|
||||
.min(10) as usize;
|
||||
let total_passes = 1 + extra_passes;
|
||||
|
||||
for (dc, endpoints) in desired_by_dc {
|
||||
if endpoints.is_empty() {
|
||||
continue;
|
||||
}
|
||||
|
||||
let mut endpoint_list: Vec<SocketAddr> = endpoints.iter().copied().collect();
|
||||
endpoint_list.sort_unstable();
|
||||
let required = Self::required_writers_for_dc(endpoint_list.len());
|
||||
let mut completed = false;
|
||||
let mut last_fresh_count = self
|
||||
.fresh_writer_count_for_endpoints(generation, endpoints)
|
||||
.await;
|
||||
|
||||
for pass_idx in 0..total_passes {
|
||||
if last_fresh_count >= required {
|
||||
completed = true;
|
||||
break;
|
||||
}
|
||||
|
||||
let missing = required.saturating_sub(last_fresh_count);
|
||||
debug!(
|
||||
dc = *dc,
|
||||
pass = pass_idx + 1,
|
||||
total_passes,
|
||||
fresh_count = last_fresh_count,
|
||||
required,
|
||||
missing,
|
||||
endpoint_count = endpoint_list.len(),
|
||||
"ME hardswap warmup pass started"
|
||||
);
|
||||
|
||||
for attempt_idx in 0..missing {
|
||||
let delay_ms = self.hardswap_warmup_connect_delay_ms();
|
||||
tokio::time::sleep(Duration::from_millis(delay_ms)).await;
|
||||
|
||||
let connected = self.connect_endpoints_round_robin(&endpoint_list, rng).await;
|
||||
debug!(
|
||||
dc = *dc,
|
||||
pass = pass_idx + 1,
|
||||
total_passes,
|
||||
attempt = attempt_idx + 1,
|
||||
delay_ms,
|
||||
connected,
|
||||
"ME hardswap warmup connect attempt finished"
|
||||
);
|
||||
}
|
||||
|
||||
last_fresh_count = self
|
||||
.fresh_writer_count_for_endpoints(generation, endpoints)
|
||||
.await;
|
||||
if last_fresh_count >= required {
|
||||
completed = true;
|
||||
info!(
|
||||
dc = *dc,
|
||||
pass = pass_idx + 1,
|
||||
total_passes,
|
||||
fresh_count = last_fresh_count,
|
||||
required,
|
||||
"ME hardswap warmup floor reached for DC"
|
||||
);
|
||||
break;
|
||||
}
|
||||
|
||||
if pass_idx + 1 < total_passes {
|
||||
let backoff_ms = self.hardswap_warmup_backoff_ms(pass_idx);
|
||||
debug!(
|
||||
dc = *dc,
|
||||
pass = pass_idx + 1,
|
||||
total_passes,
|
||||
fresh_count = last_fresh_count,
|
||||
required,
|
||||
backoff_ms,
|
||||
"ME hardswap warmup pass incomplete, delaying next pass"
|
||||
);
|
||||
tokio::time::sleep(Duration::from_millis(backoff_ms)).await;
|
||||
}
|
||||
}
|
||||
|
||||
if !completed {
|
||||
warn!(
|
||||
dc = *dc,
|
||||
fresh_count = last_fresh_count,
|
||||
required,
|
||||
endpoint_count = endpoint_list.len(),
|
||||
total_passes,
|
||||
"ME warmup stopped: unable to reach required writer floor for DC"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn zero_downtime_reinit_after_map_change(self: &Arc<Self>, rng: &SecureRandom) {
|
||||
let desired_by_dc = self.desired_dc_endpoints().await;
|
||||
if desired_by_dc.is_empty() {
|
||||
warn!("ME endpoint map is empty; skipping stale writer drain");
|
||||
return;
|
||||
}
|
||||
|
||||
let previous_generation = self.current_generation();
|
||||
let generation = self.generation.fetch_add(1, Ordering::Relaxed) + 1;
|
||||
let hardswap = self.hardswap.load(Ordering::Relaxed);
|
||||
|
||||
if hardswap {
|
||||
self.warmup_generation_for_all_dcs(rng, generation, &desired_by_dc)
|
||||
.await;
|
||||
} else {
|
||||
self.reconcile_connections(rng).await;
|
||||
}
|
||||
|
||||
let writers = self.writers.read().await;
|
||||
let active_writer_addrs: HashSet<SocketAddr> = writers
|
||||
.iter()
|
||||
.filter(|w| !w.draining.load(Ordering::Relaxed))
|
||||
.map(|w| w.addr)
|
||||
.collect();
|
||||
let min_ratio = Self::permille_to_ratio(
|
||||
self.me_pool_min_fresh_ratio_permille
|
||||
.load(Ordering::Relaxed),
|
||||
);
|
||||
let (coverage_ratio, missing_dc) = Self::coverage_ratio(&desired_by_dc, &active_writer_addrs);
|
||||
if !hardswap && coverage_ratio < min_ratio {
|
||||
warn!(
|
||||
previous_generation,
|
||||
generation,
|
||||
coverage_ratio = format_args!("{coverage_ratio:.3}"),
|
||||
min_ratio = format_args!("{min_ratio:.3}"),
|
||||
missing_dc = ?missing_dc,
|
||||
"ME reinit coverage below threshold; keeping stale writers"
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
if hardswap {
|
||||
let mut fresh_missing_dc = Vec::<(i32, usize, usize)>::new();
|
||||
for (dc, endpoints) in &desired_by_dc {
|
||||
if endpoints.is_empty() {
|
||||
continue;
|
||||
}
|
||||
let required = Self::required_writers_for_dc(endpoints.len());
|
||||
let fresh_count = writers
|
||||
.iter()
|
||||
.filter(|w| !w.draining.load(Ordering::Relaxed))
|
||||
.filter(|w| w.generation == generation)
|
||||
.filter(|w| endpoints.contains(&w.addr))
|
||||
.count();
|
||||
if fresh_count < required {
|
||||
fresh_missing_dc.push((*dc, fresh_count, required));
|
||||
}
|
||||
}
|
||||
if !fresh_missing_dc.is_empty() {
|
||||
warn!(
|
||||
previous_generation,
|
||||
generation,
|
||||
missing_dc = ?fresh_missing_dc,
|
||||
"ME hardswap pending: fresh generation coverage incomplete"
|
||||
);
|
||||
return;
|
||||
}
|
||||
} else if !missing_dc.is_empty() {
|
||||
warn!(
|
||||
missing_dc = ?missing_dc,
|
||||
// Keep stale writers alive when fresh coverage is incomplete.
|
||||
"ME reinit coverage incomplete; keeping stale writers"
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
let desired_addrs: HashSet<SocketAddr> = desired_by_dc
|
||||
.values()
|
||||
.flat_map(|set| set.iter().copied())
|
||||
.collect();
|
||||
|
||||
let stale_writer_ids: Vec<u64> = writers
|
||||
.iter()
|
||||
.filter(|w| !w.draining.load(Ordering::Relaxed))
|
||||
.filter(|w| {
|
||||
if hardswap {
|
||||
w.generation < generation
|
||||
} else {
|
||||
!desired_addrs.contains(&w.addr)
|
||||
}
|
||||
})
|
||||
.map(|w| w.id)
|
||||
.collect();
|
||||
drop(writers);
|
||||
|
||||
if stale_writer_ids.is_empty() {
|
||||
debug!("ME reinit cycle completed with no stale writers");
|
||||
return;
|
||||
}
|
||||
|
||||
let drain_timeout = self.force_close_timeout();
|
||||
let drain_timeout_secs = drain_timeout.map(|d| d.as_secs()).unwrap_or(0);
|
||||
info!(
|
||||
stale_writers = stale_writer_ids.len(),
|
||||
previous_generation,
|
||||
generation,
|
||||
hardswap,
|
||||
coverage_ratio = format_args!("{coverage_ratio:.3}"),
|
||||
min_ratio = format_args!("{min_ratio:.3}"),
|
||||
drain_timeout_secs,
|
||||
"ME reinit cycle covered; draining stale writers"
|
||||
);
|
||||
self.stats.increment_pool_swap_total();
|
||||
for writer_id in stale_writer_ids {
|
||||
self.mark_writer_draining_with_timeout(writer_id, drain_timeout, !hardswap)
|
||||
.await;
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn zero_downtime_reinit_periodic(self: &Arc<Self>, rng: &SecureRandom) {
|
||||
self.zero_downtime_reinit_after_map_change(rng).await;
|
||||
}
|
||||
}
|
||||
366
src/transport/middle_proxy/pool_writer.rs
Normal file
366
src/transport/middle_proxy/pool_writer.rs
Normal file
@@ -0,0 +1,366 @@
|
||||
use std::net::SocketAddr;
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use bytes::BytesMut;
|
||||
use rand::Rng;
|
||||
use tokio::sync::mpsc;
|
||||
use tokio_util::sync::CancellationToken;
|
||||
use tracing::{debug, info, warn};
|
||||
|
||||
use crate::crypto::SecureRandom;
|
||||
use crate::error::{ProxyError, Result};
|
||||
use crate::protocol::constants::RPC_PING_U32;
|
||||
|
||||
use super::codec::{RpcWriter, WriterCommand};
|
||||
use super::pool::{MePool, MeWriter};
|
||||
use super::reader::reader_loop;
|
||||
use super::registry::BoundConn;
|
||||
|
||||
const ME_ACTIVE_PING_SECS: u64 = 25;
|
||||
const ME_ACTIVE_PING_JITTER_SECS: i64 = 5;
|
||||
const ME_IDLE_KEEPALIVE_MAX_SECS: u64 = 5;
|
||||
|
||||
impl MePool {
|
||||
pub(crate) async fn prune_closed_writers(self: &Arc<Self>) {
|
||||
let closed_writer_ids: Vec<u64> = {
|
||||
let ws = self.writers.read().await;
|
||||
ws.iter().filter(|w| w.tx.is_closed()).map(|w| w.id).collect()
|
||||
};
|
||||
if closed_writer_ids.is_empty() {
|
||||
return;
|
||||
}
|
||||
|
||||
for writer_id in closed_writer_ids {
|
||||
if self.registry.is_writer_empty(writer_id).await {
|
||||
let _ = self.remove_writer_only(writer_id).await;
|
||||
} else {
|
||||
let _ = self.remove_writer_and_close_clients(writer_id).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) async fn connect_one(self: &Arc<Self>, addr: SocketAddr, rng: &SecureRandom) -> Result<()> {
|
||||
let secret_len = self.proxy_secret.read().await.len();
|
||||
if secret_len < 32 {
|
||||
return Err(ProxyError::Proxy("proxy-secret too short for ME auth".into()));
|
||||
}
|
||||
|
||||
let (stream, _connect_ms, upstream_egress) = self.connect_tcp(addr).await?;
|
||||
let hs = self.handshake_only(stream, addr, upstream_egress, rng).await?;
|
||||
|
||||
let writer_id = self.next_writer_id.fetch_add(1, Ordering::Relaxed);
|
||||
let generation = self.current_generation();
|
||||
let cancel = CancellationToken::new();
|
||||
let degraded = Arc::new(AtomicBool::new(false));
|
||||
let draining = Arc::new(AtomicBool::new(false));
|
||||
let draining_started_at_epoch_secs = Arc::new(AtomicU64::new(0));
|
||||
let allow_drain_fallback = Arc::new(AtomicBool::new(false));
|
||||
let (tx, mut rx) = mpsc::channel::<WriterCommand>(4096);
|
||||
let mut rpc_writer = RpcWriter {
|
||||
writer: hs.wr,
|
||||
key: hs.write_key,
|
||||
iv: hs.write_iv,
|
||||
seq_no: 0,
|
||||
crc_mode: hs.crc_mode,
|
||||
};
|
||||
let cancel_wr = cancel.clone();
|
||||
tokio::spawn(async move {
|
||||
loop {
|
||||
tokio::select! {
|
||||
cmd = rx.recv() => {
|
||||
match cmd {
|
||||
Some(WriterCommand::Data(payload)) => {
|
||||
if rpc_writer.send(&payload).await.is_err() { break; }
|
||||
}
|
||||
Some(WriterCommand::DataAndFlush(payload)) => {
|
||||
if rpc_writer.send_and_flush(&payload).await.is_err() { break; }
|
||||
}
|
||||
Some(WriterCommand::Close) | None => break,
|
||||
}
|
||||
}
|
||||
_ = cancel_wr.cancelled() => break,
|
||||
}
|
||||
}
|
||||
});
|
||||
let writer = MeWriter {
|
||||
id: writer_id,
|
||||
addr,
|
||||
generation,
|
||||
tx: tx.clone(),
|
||||
cancel: cancel.clone(),
|
||||
degraded: degraded.clone(),
|
||||
draining: draining.clone(),
|
||||
draining_started_at_epoch_secs: draining_started_at_epoch_secs.clone(),
|
||||
allow_drain_fallback: allow_drain_fallback.clone(),
|
||||
};
|
||||
self.writers.write().await.push(writer.clone());
|
||||
self.conn_count.fetch_add(1, Ordering::Relaxed);
|
||||
self.writer_available.notify_one();
|
||||
|
||||
let reg = self.registry.clone();
|
||||
let writers_arc = self.writers_arc();
|
||||
let ping_tracker = self.ping_tracker.clone();
|
||||
let ping_tracker_reader = ping_tracker.clone();
|
||||
let rtt_stats = self.rtt_stats.clone();
|
||||
let stats_reader = self.stats.clone();
|
||||
let stats_ping = self.stats.clone();
|
||||
let pool = Arc::downgrade(self);
|
||||
let cancel_ping = cancel.clone();
|
||||
let tx_ping = tx.clone();
|
||||
let ping_tracker_ping = ping_tracker.clone();
|
||||
let cleanup_done = Arc::new(AtomicBool::new(false));
|
||||
let cleanup_for_reader = cleanup_done.clone();
|
||||
let cleanup_for_ping = cleanup_done.clone();
|
||||
let keepalive_enabled = self.me_keepalive_enabled;
|
||||
let keepalive_interval = self.me_keepalive_interval;
|
||||
let keepalive_jitter = self.me_keepalive_jitter;
|
||||
let cancel_reader_token = cancel.clone();
|
||||
let cancel_ping_token = cancel_ping.clone();
|
||||
|
||||
tokio::spawn(async move {
|
||||
let res = reader_loop(
|
||||
hs.rd,
|
||||
hs.read_key,
|
||||
hs.read_iv,
|
||||
hs.crc_mode,
|
||||
reg.clone(),
|
||||
BytesMut::new(),
|
||||
BytesMut::new(),
|
||||
tx.clone(),
|
||||
ping_tracker_reader,
|
||||
rtt_stats.clone(),
|
||||
stats_reader,
|
||||
writer_id,
|
||||
degraded.clone(),
|
||||
cancel_reader_token.clone(),
|
||||
)
|
||||
.await;
|
||||
if let Some(pool) = pool.upgrade()
|
||||
&& cleanup_for_reader
|
||||
.compare_exchange(false, true, Ordering::AcqRel, Ordering::Relaxed)
|
||||
.is_ok()
|
||||
{
|
||||
pool.remove_writer_and_close_clients(writer_id).await;
|
||||
}
|
||||
if let Err(e) = res {
|
||||
warn!(error = %e, "ME reader ended");
|
||||
}
|
||||
let mut ws = writers_arc.write().await;
|
||||
ws.retain(|w| w.id != writer_id);
|
||||
info!(remaining = ws.len(), "Dead ME writer removed from pool");
|
||||
});
|
||||
|
||||
let pool_ping = Arc::downgrade(self);
|
||||
tokio::spawn(async move {
|
||||
let mut ping_id: i64 = rand::random::<i64>();
|
||||
let idle_interval_cap = Duration::from_secs(ME_IDLE_KEEPALIVE_MAX_SECS);
|
||||
// Per-writer jittered start to avoid phase sync.
|
||||
let startup_jitter = if keepalive_enabled {
|
||||
let mut interval = keepalive_interval;
|
||||
if let Some(pool) = pool_ping.upgrade() {
|
||||
if pool.registry.is_writer_empty(writer_id).await {
|
||||
interval = interval.min(idle_interval_cap);
|
||||
}
|
||||
} else {
|
||||
return;
|
||||
}
|
||||
let jitter_cap_ms = interval.as_millis() / 2;
|
||||
let effective_jitter_ms = keepalive_jitter.as_millis().min(jitter_cap_ms).max(1);
|
||||
Duration::from_millis(rand::rng().random_range(0..=effective_jitter_ms as u64))
|
||||
} else {
|
||||
let jitter = rand::rng().random_range(-ME_ACTIVE_PING_JITTER_SECS..=ME_ACTIVE_PING_JITTER_SECS);
|
||||
let wait = (ME_ACTIVE_PING_SECS as i64 + jitter).max(5) as u64;
|
||||
Duration::from_secs(wait)
|
||||
};
|
||||
tokio::select! {
|
||||
_ = cancel_ping_token.cancelled() => return,
|
||||
_ = tokio::time::sleep(startup_jitter) => {}
|
||||
}
|
||||
loop {
|
||||
let wait = if keepalive_enabled {
|
||||
let mut interval = keepalive_interval;
|
||||
if let Some(pool) = pool_ping.upgrade() {
|
||||
if pool.registry.is_writer_empty(writer_id).await {
|
||||
interval = interval.min(idle_interval_cap);
|
||||
}
|
||||
} else {
|
||||
break;
|
||||
}
|
||||
let jitter_cap_ms = interval.as_millis() / 2;
|
||||
let effective_jitter_ms = keepalive_jitter.as_millis().min(jitter_cap_ms).max(1);
|
||||
interval + Duration::from_millis(rand::rng().random_range(0..=effective_jitter_ms as u64))
|
||||
} else {
|
||||
let jitter = rand::rng().random_range(-ME_ACTIVE_PING_JITTER_SECS..=ME_ACTIVE_PING_JITTER_SECS);
|
||||
let secs = (ME_ACTIVE_PING_SECS as i64 + jitter).max(5) as u64;
|
||||
Duration::from_secs(secs)
|
||||
};
|
||||
tokio::select! {
|
||||
_ = cancel_ping_token.cancelled() => {
|
||||
break;
|
||||
}
|
||||
_ = tokio::time::sleep(wait) => {}
|
||||
}
|
||||
let sent_id = ping_id;
|
||||
let mut p = Vec::with_capacity(12);
|
||||
p.extend_from_slice(&RPC_PING_U32.to_le_bytes());
|
||||
p.extend_from_slice(&sent_id.to_le_bytes());
|
||||
{
|
||||
let mut tracker = ping_tracker_ping.lock().await;
|
||||
let before = tracker.len();
|
||||
tracker.retain(|_, (ts, _)| ts.elapsed() < Duration::from_secs(120));
|
||||
let expired = before.saturating_sub(tracker.len());
|
||||
if expired > 0 {
|
||||
stats_ping.increment_me_keepalive_timeout_by(expired as u64);
|
||||
}
|
||||
tracker.insert(sent_id, (std::time::Instant::now(), writer_id));
|
||||
}
|
||||
ping_id = ping_id.wrapping_add(1);
|
||||
stats_ping.increment_me_keepalive_sent();
|
||||
if tx_ping.send(WriterCommand::DataAndFlush(p)).await.is_err() {
|
||||
stats_ping.increment_me_keepalive_failed();
|
||||
debug!("ME ping failed, removing dead writer");
|
||||
cancel_ping.cancel();
|
||||
if let Some(pool) = pool_ping.upgrade()
|
||||
&& cleanup_for_ping
|
||||
.compare_exchange(false, true, Ordering::AcqRel, Ordering::Relaxed)
|
||||
.is_ok()
|
||||
{
|
||||
pool.remove_writer_and_close_clients(writer_id).await;
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(crate) async fn remove_writer_and_close_clients(self: &Arc<Self>, writer_id: u64) {
|
||||
let conns = self.remove_writer_only(writer_id).await;
|
||||
for bound in conns {
|
||||
let _ = self.registry.route(bound.conn_id, super::MeResponse::Close).await;
|
||||
let _ = self.registry.unregister(bound.conn_id).await;
|
||||
}
|
||||
}
|
||||
|
||||
async fn remove_writer_only(self: &Arc<Self>, writer_id: u64) -> Vec<BoundConn> {
|
||||
let mut close_tx: Option<mpsc::Sender<WriterCommand>> = None;
|
||||
let mut removed_addr: Option<SocketAddr> = None;
|
||||
let mut trigger_refill = false;
|
||||
{
|
||||
let mut ws = self.writers.write().await;
|
||||
if let Some(pos) = ws.iter().position(|w| w.id == writer_id) {
|
||||
let w = ws.remove(pos);
|
||||
let was_draining = w.draining.load(Ordering::Relaxed);
|
||||
if was_draining {
|
||||
self.stats.decrement_pool_drain_active();
|
||||
}
|
||||
self.stats.increment_me_writer_removed_total();
|
||||
w.cancel.cancel();
|
||||
removed_addr = Some(w.addr);
|
||||
trigger_refill = !was_draining;
|
||||
if trigger_refill {
|
||||
self.stats.increment_me_writer_removed_unexpected_total();
|
||||
}
|
||||
close_tx = Some(w.tx.clone());
|
||||
self.conn_count.fetch_sub(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
if let Some(tx) = close_tx {
|
||||
let _ = tx.send(WriterCommand::Close).await;
|
||||
}
|
||||
if trigger_refill
|
||||
&& let Some(addr) = removed_addr
|
||||
{
|
||||
self.trigger_immediate_refill(addr);
|
||||
}
|
||||
self.rtt_stats.lock().await.remove(&writer_id);
|
||||
self.registry.writer_lost(writer_id).await
|
||||
}
|
||||
|
||||
pub(crate) async fn mark_writer_draining_with_timeout(
|
||||
self: &Arc<Self>,
|
||||
writer_id: u64,
|
||||
timeout: Option<Duration>,
|
||||
allow_drain_fallback: bool,
|
||||
) {
|
||||
let timeout = timeout.filter(|d| !d.is_zero());
|
||||
let found = {
|
||||
let mut ws = self.writers.write().await;
|
||||
if let Some(w) = ws.iter_mut().find(|w| w.id == writer_id) {
|
||||
let already_draining = w.draining.swap(true, Ordering::Relaxed);
|
||||
w.allow_drain_fallback
|
||||
.store(allow_drain_fallback, Ordering::Relaxed);
|
||||
w.draining_started_at_epoch_secs
|
||||
.store(Self::now_epoch_secs(), Ordering::Relaxed);
|
||||
if !already_draining {
|
||||
self.stats.increment_pool_drain_active();
|
||||
}
|
||||
w.draining.store(true, Ordering::Relaxed);
|
||||
true
|
||||
} else {
|
||||
false
|
||||
}
|
||||
};
|
||||
|
||||
if !found {
|
||||
return;
|
||||
}
|
||||
|
||||
let timeout_secs = timeout.map(|d| d.as_secs()).unwrap_or(0);
|
||||
debug!(
|
||||
writer_id,
|
||||
timeout_secs,
|
||||
allow_drain_fallback,
|
||||
"ME writer marked draining"
|
||||
);
|
||||
|
||||
let pool = Arc::downgrade(self);
|
||||
tokio::spawn(async move {
|
||||
let deadline = timeout.map(|t| Instant::now() + t);
|
||||
while let Some(p) = pool.upgrade() {
|
||||
if let Some(deadline_at) = deadline
|
||||
&& Instant::now() >= deadline_at
|
||||
{
|
||||
warn!(writer_id, "Drain timeout, force-closing");
|
||||
p.stats.increment_pool_force_close_total();
|
||||
let _ = p.remove_writer_and_close_clients(writer_id).await;
|
||||
break;
|
||||
}
|
||||
if p.registry.is_writer_empty(writer_id).await {
|
||||
let _ = p.remove_writer_only(writer_id).await;
|
||||
break;
|
||||
}
|
||||
tokio::time::sleep(Duration::from_secs(1)).await;
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
pub(crate) async fn mark_writer_draining(self: &Arc<Self>, writer_id: u64) {
|
||||
self.mark_writer_draining_with_timeout(writer_id, Some(Duration::from_secs(300)), false)
|
||||
.await;
|
||||
}
|
||||
|
||||
pub(super) fn writer_accepts_new_binding(&self, writer: &MeWriter) -> bool {
|
||||
if !writer.draining.load(Ordering::Relaxed) {
|
||||
return true;
|
||||
}
|
||||
if !writer.allow_drain_fallback.load(Ordering::Relaxed) {
|
||||
return false;
|
||||
}
|
||||
|
||||
let ttl_secs = self.me_pool_drain_ttl_secs.load(Ordering::Relaxed);
|
||||
if ttl_secs == 0 {
|
||||
return true;
|
||||
}
|
||||
|
||||
let started = writer.draining_started_at_epoch_secs.load(Ordering::Relaxed);
|
||||
if started == 0 {
|
||||
return false;
|
||||
}
|
||||
|
||||
Self::now_epoch_secs().saturating_sub(started) <= ttl_secs
|
||||
}
|
||||
}
|
||||
@@ -124,7 +124,14 @@ pub(crate) async fn reader_loop(
|
||||
match routed {
|
||||
RouteResult::NoConn => stats.increment_me_route_drop_no_conn(),
|
||||
RouteResult::ChannelClosed => stats.increment_me_route_drop_channel_closed(),
|
||||
RouteResult::QueueFull => stats.increment_me_route_drop_queue_full(),
|
||||
RouteResult::QueueFullBase => {
|
||||
stats.increment_me_route_drop_queue_full();
|
||||
stats.increment_me_route_drop_queue_full_base();
|
||||
}
|
||||
RouteResult::QueueFullHigh => {
|
||||
stats.increment_me_route_drop_queue_full();
|
||||
stats.increment_me_route_drop_queue_full_high();
|
||||
}
|
||||
RouteResult::Routed => {}
|
||||
}
|
||||
reg.unregister(cid).await;
|
||||
@@ -140,7 +147,14 @@ pub(crate) async fn reader_loop(
|
||||
match routed {
|
||||
RouteResult::NoConn => stats.increment_me_route_drop_no_conn(),
|
||||
RouteResult::ChannelClosed => stats.increment_me_route_drop_channel_closed(),
|
||||
RouteResult::QueueFull => stats.increment_me_route_drop_queue_full(),
|
||||
RouteResult::QueueFullBase => {
|
||||
stats.increment_me_route_drop_queue_full();
|
||||
stats.increment_me_route_drop_queue_full_base();
|
||||
}
|
||||
RouteResult::QueueFullHigh => {
|
||||
stats.increment_me_route_drop_queue_full();
|
||||
stats.increment_me_route_drop_queue_full_high();
|
||||
}
|
||||
RouteResult::Routed => {}
|
||||
}
|
||||
reg.unregister(cid).await;
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::net::SocketAddr;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::sync::atomic::{AtomicU8, AtomicU64, Ordering};
|
||||
use std::time::Duration;
|
||||
|
||||
use tokio::sync::{mpsc, RwLock};
|
||||
@@ -10,17 +10,21 @@ use super::codec::WriterCommand;
|
||||
use super::MeResponse;
|
||||
|
||||
const ROUTE_CHANNEL_CAPACITY: usize = 4096;
|
||||
const ROUTE_BACKPRESSURE_TIMEOUT: Duration = Duration::from_millis(25);
|
||||
const ROUTE_BACKPRESSURE_BASE_TIMEOUT_MS: u64 = 25;
|
||||
const ROUTE_BACKPRESSURE_HIGH_TIMEOUT_MS: u64 = 120;
|
||||
const ROUTE_BACKPRESSURE_HIGH_WATERMARK_PCT: u8 = 80;
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum RouteResult {
|
||||
Routed,
|
||||
NoConn,
|
||||
ChannelClosed,
|
||||
QueueFull,
|
||||
QueueFullBase,
|
||||
QueueFullHigh,
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
#[allow(dead_code)]
|
||||
pub struct ConnMeta {
|
||||
pub target_dc: i16,
|
||||
pub client_addr: SocketAddr,
|
||||
@@ -29,6 +33,7 @@ pub struct ConnMeta {
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
#[allow(dead_code)]
|
||||
pub struct BoundConn {
|
||||
pub conn_id: u64,
|
||||
pub meta: ConnMeta,
|
||||
@@ -63,6 +68,9 @@ impl RegistryInner {
|
||||
pub struct ConnRegistry {
|
||||
inner: RwLock<RegistryInner>,
|
||||
next_id: AtomicU64,
|
||||
route_backpressure_base_timeout_ms: AtomicU64,
|
||||
route_backpressure_high_timeout_ms: AtomicU64,
|
||||
route_backpressure_high_watermark_pct: AtomicU8,
|
||||
}
|
||||
|
||||
impl ConnRegistry {
|
||||
@@ -71,9 +79,35 @@ impl ConnRegistry {
|
||||
Self {
|
||||
inner: RwLock::new(RegistryInner::new()),
|
||||
next_id: AtomicU64::new(start),
|
||||
route_backpressure_base_timeout_ms: AtomicU64::new(
|
||||
ROUTE_BACKPRESSURE_BASE_TIMEOUT_MS,
|
||||
),
|
||||
route_backpressure_high_timeout_ms: AtomicU64::new(
|
||||
ROUTE_BACKPRESSURE_HIGH_TIMEOUT_MS,
|
||||
),
|
||||
route_backpressure_high_watermark_pct: AtomicU8::new(
|
||||
ROUTE_BACKPRESSURE_HIGH_WATERMARK_PCT,
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn update_route_backpressure_policy(
|
||||
&self,
|
||||
base_timeout_ms: u64,
|
||||
high_timeout_ms: u64,
|
||||
high_watermark_pct: u8,
|
||||
) {
|
||||
let base = base_timeout_ms.max(1);
|
||||
let high = high_timeout_ms.max(base);
|
||||
let watermark = high_watermark_pct.clamp(1, 100);
|
||||
self.route_backpressure_base_timeout_ms
|
||||
.store(base, Ordering::Relaxed);
|
||||
self.route_backpressure_high_timeout_ms
|
||||
.store(high, Ordering::Relaxed);
|
||||
self.route_backpressure_high_watermark_pct
|
||||
.store(watermark, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
pub async fn register(&self) -> (u64, mpsc::Receiver<MeResponse>) {
|
||||
let id = self.next_id.fetch_add(1, Ordering::Relaxed);
|
||||
let (tx, rx) = mpsc::channel(ROUTE_CHANNEL_CAPACITY);
|
||||
@@ -110,10 +144,40 @@ impl ConnRegistry {
|
||||
Err(TrySendError::Closed(_)) => RouteResult::ChannelClosed,
|
||||
Err(TrySendError::Full(resp)) => {
|
||||
// Absorb short bursts without dropping/closing the session immediately.
|
||||
match tokio::time::timeout(ROUTE_BACKPRESSURE_TIMEOUT, tx.send(resp)).await {
|
||||
let base_timeout_ms =
|
||||
self.route_backpressure_base_timeout_ms.load(Ordering::Relaxed).max(1);
|
||||
let high_timeout_ms = self
|
||||
.route_backpressure_high_timeout_ms
|
||||
.load(Ordering::Relaxed)
|
||||
.max(base_timeout_ms);
|
||||
let high_watermark_pct = self
|
||||
.route_backpressure_high_watermark_pct
|
||||
.load(Ordering::Relaxed)
|
||||
.clamp(1, 100);
|
||||
let used = ROUTE_CHANNEL_CAPACITY.saturating_sub(tx.capacity());
|
||||
let used_pct = if ROUTE_CHANNEL_CAPACITY == 0 {
|
||||
100
|
||||
} else {
|
||||
(used.saturating_mul(100) / ROUTE_CHANNEL_CAPACITY) as u8
|
||||
};
|
||||
let high_profile = used_pct >= high_watermark_pct;
|
||||
let timeout_ms = if high_profile {
|
||||
high_timeout_ms
|
||||
} else {
|
||||
base_timeout_ms
|
||||
};
|
||||
let timeout_dur = Duration::from_millis(timeout_ms);
|
||||
|
||||
match tokio::time::timeout(timeout_dur, tx.send(resp)).await {
|
||||
Ok(Ok(())) => RouteResult::Routed,
|
||||
Ok(Err(_)) => RouteResult::ChannelClosed,
|
||||
Err(_) => RouteResult::QueueFull,
|
||||
Err(_) => {
|
||||
if high_profile {
|
||||
RouteResult::QueueFullHigh
|
||||
} else {
|
||||
RouteResult::QueueFullBase
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -167,6 +231,7 @@ impl ConnRegistry {
|
||||
out
|
||||
}
|
||||
|
||||
#[allow(dead_code)]
|
||||
pub async fn get_meta(&self, conn_id: u64) -> Option<ConnMeta> {
|
||||
let inner = self.inner.read().await;
|
||||
inner.meta.get(&conn_id).cloned()
|
||||
|
||||
@@ -1,50 +1,87 @@
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::Ordering;
|
||||
use std::time::Duration;
|
||||
|
||||
use tokio::sync::watch;
|
||||
use tracing::{info, warn};
|
||||
|
||||
use crate::config::ProxyConfig;
|
||||
use crate::crypto::SecureRandom;
|
||||
|
||||
use super::MePool;
|
||||
|
||||
/// Periodically refresh ME connections to avoid long-lived degradation.
|
||||
pub async fn me_rotation_task(pool: Arc<MePool>, rng: Arc<SecureRandom>, interval: Duration) {
|
||||
let interval = interval.max(Duration::from_secs(600));
|
||||
/// Periodically reinitialize ME generations and swap them after full warmup.
|
||||
pub async fn me_rotation_task(
|
||||
pool: Arc<MePool>,
|
||||
rng: Arc<SecureRandom>,
|
||||
mut config_rx: watch::Receiver<Arc<ProxyConfig>>,
|
||||
) {
|
||||
let mut interval_secs = config_rx
|
||||
.borrow()
|
||||
.general
|
||||
.effective_me_reinit_every_secs()
|
||||
.max(1);
|
||||
let mut interval = Duration::from_secs(interval_secs);
|
||||
let mut next_tick = tokio::time::Instant::now() + interval;
|
||||
|
||||
info!(interval_secs, "ME periodic reinit task started");
|
||||
|
||||
loop {
|
||||
tokio::time::sleep(interval).await;
|
||||
let sleep = tokio::time::sleep_until(next_tick);
|
||||
tokio::pin!(sleep);
|
||||
|
||||
let candidate = {
|
||||
let ws = pool.writers.read().await;
|
||||
if ws.is_empty() {
|
||||
None
|
||||
} else {
|
||||
let idx = (pool.rr.load(std::sync::atomic::Ordering::Relaxed) as usize) % ws.len();
|
||||
ws.get(idx).cloned()
|
||||
}
|
||||
};
|
||||
|
||||
let Some(w) = candidate else {
|
||||
continue;
|
||||
};
|
||||
|
||||
info!(addr = %w.addr, writer_id = w.id, "Rotating ME connection");
|
||||
match pool.connect_one(w.addr, rng.as_ref()).await {
|
||||
Ok(()) => {
|
||||
tokio::time::sleep(Duration::from_secs(2)).await;
|
||||
let ws = pool.writers.read().await;
|
||||
let new_alive = ws.iter().any(|nw|
|
||||
nw.id != w.id && nw.addr == w.addr && !nw.degraded.load(Ordering::Relaxed) && !nw.draining.load(Ordering::Relaxed)
|
||||
);
|
||||
drop(ws);
|
||||
if new_alive {
|
||||
pool.mark_writer_draining(w.id).await;
|
||||
} else {
|
||||
warn!(addr = %w.addr, writer_id = w.id, "New writer died, keeping old");
|
||||
tokio::select! {
|
||||
_ = &mut sleep => {
|
||||
pool.zero_downtime_reinit_periodic(rng.as_ref()).await;
|
||||
let refreshed_secs = config_rx
|
||||
.borrow()
|
||||
.general
|
||||
.effective_me_reinit_every_secs()
|
||||
.max(1);
|
||||
if refreshed_secs != interval_secs {
|
||||
info!(
|
||||
old_me_reinit_every_secs = interval_secs,
|
||||
new_me_reinit_every_secs = refreshed_secs,
|
||||
"ME periodic reinit interval changed"
|
||||
);
|
||||
interval_secs = refreshed_secs;
|
||||
interval = Duration::from_secs(interval_secs);
|
||||
}
|
||||
next_tick = tokio::time::Instant::now() + interval;
|
||||
}
|
||||
Err(e) => {
|
||||
warn!(addr = %w.addr, writer_id = w.id, error = %e, "ME rotation connect failed");
|
||||
changed = config_rx.changed() => {
|
||||
if changed.is_err() {
|
||||
warn!("ME periodic reinit task stopped: config channel closed");
|
||||
break;
|
||||
}
|
||||
let new_secs = config_rx
|
||||
.borrow()
|
||||
.general
|
||||
.effective_me_reinit_every_secs()
|
||||
.max(1);
|
||||
if new_secs == interval_secs {
|
||||
continue;
|
||||
}
|
||||
|
||||
if new_secs < interval_secs {
|
||||
info!(
|
||||
old_me_reinit_every_secs = interval_secs,
|
||||
new_me_reinit_every_secs = new_secs,
|
||||
"ME periodic reinit interval decreased, running immediate reinit"
|
||||
);
|
||||
interval_secs = new_secs;
|
||||
interval = Duration::from_secs(interval_secs);
|
||||
pool.zero_downtime_reinit_periodic(rng.as_ref()).await;
|
||||
next_tick = tokio::time::Instant::now() + interval;
|
||||
} else {
|
||||
info!(
|
||||
old_me_reinit_every_secs = interval_secs,
|
||||
new_me_reinit_every_secs = new_secs,
|
||||
"ME periodic reinit interval increased"
|
||||
);
|
||||
interval_secs = new_secs;
|
||||
interval = Duration::from_secs(interval_secs);
|
||||
next_tick = tokio::time::Instant::now() + interval;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,17 +1,45 @@
|
||||
use std::time::Duration;
|
||||
|
||||
use tracing::{debug, info, warn};
|
||||
use std::time::SystemTime;
|
||||
use httpdate;
|
||||
|
||||
use crate::error::{ProxyError, Result};
|
||||
|
||||
pub const PROXY_SECRET_MIN_LEN: usize = 32;
|
||||
|
||||
pub(super) fn validate_proxy_secret_len(data_len: usize, max_len: usize) -> Result<()> {
|
||||
if max_len < PROXY_SECRET_MIN_LEN {
|
||||
return Err(ProxyError::Proxy(format!(
|
||||
"proxy-secret max length is invalid: {} bytes (must be >= {})",
|
||||
max_len,
|
||||
PROXY_SECRET_MIN_LEN
|
||||
)));
|
||||
}
|
||||
|
||||
if data_len < PROXY_SECRET_MIN_LEN {
|
||||
return Err(ProxyError::Proxy(format!(
|
||||
"proxy-secret too short: {} bytes (need >= {})",
|
||||
data_len,
|
||||
PROXY_SECRET_MIN_LEN
|
||||
)));
|
||||
}
|
||||
|
||||
if data_len > max_len {
|
||||
return Err(ProxyError::Proxy(format!(
|
||||
"proxy-secret too long: {} bytes (limit = {})",
|
||||
data_len,
|
||||
max_len
|
||||
)));
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Fetch Telegram proxy-secret binary.
|
||||
pub async fn fetch_proxy_secret(cache_path: Option<&str>) -> Result<Vec<u8>> {
|
||||
pub async fn fetch_proxy_secret(cache_path: Option<&str>, max_len: usize) -> Result<Vec<u8>> {
|
||||
let cache = cache_path.unwrap_or("proxy-secret");
|
||||
|
||||
// 1) Try fresh download first.
|
||||
match download_proxy_secret().await {
|
||||
match download_proxy_secret_with_max_len(max_len).await {
|
||||
Ok(data) => {
|
||||
if let Err(e) = tokio::fs::write(cache, &data).await {
|
||||
warn!(error = %e, "Failed to cache proxy-secret (non-fatal)");
|
||||
@@ -26,9 +54,9 @@ pub async fn fetch_proxy_secret(cache_path: Option<&str>) -> Result<Vec<u8>> {
|
||||
}
|
||||
}
|
||||
|
||||
// 2) Fallback to cache/file regardless of age; require len>=32.
|
||||
// 2) Fallback to cache/file regardless of age; require len in bounds.
|
||||
match tokio::fs::read(cache).await {
|
||||
Ok(data) if data.len() >= 32 => {
|
||||
Ok(data) if validate_proxy_secret_len(data.len(), max_len).is_ok() => {
|
||||
let age_hours = tokio::fs::metadata(cache)
|
||||
.await
|
||||
.ok()
|
||||
@@ -43,17 +71,14 @@ pub async fn fetch_proxy_secret(cache_path: Option<&str>) -> Result<Vec<u8>> {
|
||||
);
|
||||
Ok(data)
|
||||
}
|
||||
Ok(data) => Err(ProxyError::Proxy(format!(
|
||||
"Cached proxy-secret too short: {} bytes (need >= 32)",
|
||||
data.len()
|
||||
))),
|
||||
Ok(data) => validate_proxy_secret_len(data.len(), max_len).map(|_| data),
|
||||
Err(e) => Err(ProxyError::Proxy(format!(
|
||||
"Failed to read proxy-secret cache after download failure: {e}"
|
||||
))),
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn download_proxy_secret() -> Result<Vec<u8>> {
|
||||
pub async fn download_proxy_secret_with_max_len(max_len: usize) -> Result<Vec<u8>> {
|
||||
let resp = reqwest::get("https://core.telegram.org/getProxySecret")
|
||||
.await
|
||||
.map_err(|e| ProxyError::Proxy(format!("Failed to download proxy-secret: {e}")))?;
|
||||
@@ -65,20 +90,18 @@ pub async fn download_proxy_secret() -> Result<Vec<u8>> {
|
||||
)));
|
||||
}
|
||||
|
||||
if let Some(date) = resp.headers().get(reqwest::header::DATE) {
|
||||
if let Ok(date_str) = date.to_str() {
|
||||
if let Ok(server_time) = httpdate::parse_http_date(date_str) {
|
||||
if let Ok(skew) = SystemTime::now().duration_since(server_time).or_else(|e| {
|
||||
server_time.duration_since(SystemTime::now()).map_err(|_| e)
|
||||
}) {
|
||||
let skew_secs = skew.as_secs();
|
||||
if skew_secs > 60 {
|
||||
warn!(skew_secs, "Time skew >60s detected from proxy-secret Date header");
|
||||
} else if skew_secs > 30 {
|
||||
warn!(skew_secs, "Time skew >30s detected from proxy-secret Date header");
|
||||
}
|
||||
}
|
||||
}
|
||||
if let Some(date) = resp.headers().get(reqwest::header::DATE)
|
||||
&& let Ok(date_str) = date.to_str()
|
||||
&& let Ok(server_time) = httpdate::parse_http_date(date_str)
|
||||
&& let Ok(skew) = SystemTime::now().duration_since(server_time).or_else(|e| {
|
||||
server_time.duration_since(SystemTime::now()).map_err(|_| e)
|
||||
})
|
||||
{
|
||||
let skew_secs = skew.as_secs();
|
||||
if skew_secs > 60 {
|
||||
warn!(skew_secs, "Time skew >60s detected from proxy-secret Date header");
|
||||
} else if skew_secs > 30 {
|
||||
warn!(skew_secs, "Time skew >30s detected from proxy-secret Date header");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -88,12 +111,7 @@ pub async fn download_proxy_secret() -> Result<Vec<u8>> {
|
||||
.map_err(|e| ProxyError::Proxy(format!("Read proxy-secret body: {e}")))?
|
||||
.to_vec();
|
||||
|
||||
if data.len() < 32 {
|
||||
return Err(ProxyError::Proxy(format!(
|
||||
"proxy-secret too short: {} bytes (need >= 32)",
|
||||
data.len()
|
||||
)));
|
||||
}
|
||||
validate_proxy_secret_len(data.len(), max_len)?;
|
||||
|
||||
info!(len = data.len(), "Downloaded proxy-secret OK");
|
||||
Ok(data)
|
||||
|
||||
@@ -242,10 +242,10 @@ impl MePool {
|
||||
}
|
||||
if preferred.is_empty() {
|
||||
let def = self.default_dc.load(Ordering::Relaxed);
|
||||
if def != 0 {
|
||||
if let Some(v) = map_guard.get(&def) {
|
||||
preferred.extend(v.iter().map(|(ip, port)| SocketAddr::new(*ip, *port)));
|
||||
}
|
||||
if def != 0
|
||||
&& let Some(v) = map_guard.get(&def)
|
||||
{
|
||||
preferred.extend(v.iter().map(|(ip, port)| SocketAddr::new(*ip, *port)));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -267,7 +267,7 @@ impl MePool {
|
||||
if !self.writer_accepts_new_binding(w) {
|
||||
continue;
|
||||
}
|
||||
if preferred.iter().any(|p| *p == w.addr) {
|
||||
if preferred.contains(&w.addr) {
|
||||
out.push(idx);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -6,9 +6,13 @@ pub mod socket;
|
||||
pub mod socks;
|
||||
pub mod upstream;
|
||||
|
||||
#[allow(unused_imports)]
|
||||
pub use pool::ConnectionPool;
|
||||
#[allow(unused_imports)]
|
||||
pub use proxy_protocol::{ProxyProtocolInfo, parse_proxy_protocol};
|
||||
pub use socket::*;
|
||||
#[allow(unused_imports)]
|
||||
pub use socks::*;
|
||||
pub use upstream::{DcPingResult, StartupPingResult, UpstreamManager};
|
||||
#[allow(unused_imports)]
|
||||
pub use upstream::{DcPingResult, StartupPingResult, UpstreamEgressInfo, UpstreamManager, UpstreamRouteKind};
|
||||
pub mod middle_proxy;
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
//! Connection Pool
|
||||
|
||||
#![allow(dead_code)]
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::net::SocketAddr;
|
||||
use std::sync::Arc;
|
||||
@@ -8,7 +10,7 @@ use tokio::net::TcpStream;
|
||||
use tokio::sync::Mutex;
|
||||
use tokio::time::timeout;
|
||||
use parking_lot::RwLock;
|
||||
use tracing::{debug, warn};
|
||||
use tracing::debug;
|
||||
use crate::error::{ProxyError, Result};
|
||||
use super::socket::configure_tcp_socket;
|
||||
|
||||
|
||||
@@ -28,6 +28,7 @@ mod address_family {
|
||||
|
||||
/// Information extracted from PROXY protocol header
|
||||
#[derive(Debug, Clone)]
|
||||
#[allow(dead_code)]
|
||||
pub struct ProxyProtocolInfo {
|
||||
/// Source (client) address
|
||||
pub src_addr: SocketAddr,
|
||||
@@ -37,6 +38,7 @@ pub struct ProxyProtocolInfo {
|
||||
pub version: u8,
|
||||
}
|
||||
|
||||
#[allow(dead_code)]
|
||||
impl ProxyProtocolInfo {
|
||||
/// Create info with just source address
|
||||
pub fn new(src_addr: SocketAddr) -> Self {
|
||||
@@ -289,6 +291,12 @@ pub struct ProxyProtocolV2Builder {
|
||||
dst: Option<SocketAddr>,
|
||||
}
|
||||
|
||||
impl Default for ProxyProtocolV2Builder {
|
||||
fn default() -> Self {
|
||||
Self::new()
|
||||
}
|
||||
}
|
||||
|
||||
impl ProxyProtocolV2Builder {
|
||||
pub fn new() -> Self {
|
||||
Self { src: None, dst: None }
|
||||
|
||||
@@ -10,6 +10,7 @@ use socket2::{Socket, TcpKeepalive, Domain, Type, Protocol};
|
||||
use tracing::debug;
|
||||
|
||||
/// Configure TCP socket with recommended settings for proxy use
|
||||
#[allow(dead_code)]
|
||||
pub fn configure_tcp_socket(
|
||||
stream: &TcpStream,
|
||||
keepalive: bool,
|
||||
@@ -82,6 +83,7 @@ pub fn configure_client_socket(
|
||||
}
|
||||
|
||||
/// Set socket to send RST on close (for masking)
|
||||
#[allow(dead_code)]
|
||||
pub fn set_linger_zero(stream: &TcpStream) -> Result<()> {
|
||||
let socket = socket2::SockRef::from(stream);
|
||||
socket.set_linger(Some(Duration::ZERO))?;
|
||||
@@ -89,6 +91,7 @@ pub fn set_linger_zero(stream: &TcpStream) -> Result<()> {
|
||||
}
|
||||
|
||||
/// Create a new TCP socket for outgoing connections
|
||||
#[allow(dead_code)]
|
||||
pub fn create_outgoing_socket(addr: SocketAddr) -> Result<Socket> {
|
||||
create_outgoing_socket_bound(addr, None)
|
||||
}
|
||||
@@ -120,6 +123,7 @@ pub fn create_outgoing_socket_bound(addr: SocketAddr, bind_addr: Option<IpAddr>)
|
||||
|
||||
|
||||
/// Get local address of a socket
|
||||
#[allow(dead_code)]
|
||||
pub fn get_local_addr(stream: &TcpStream) -> Option<SocketAddr> {
|
||||
stream.local_addr().ok()
|
||||
}
|
||||
@@ -132,17 +136,17 @@ pub fn resolve_interface_ip(name: &str, want_ipv6: bool) -> Option<IpAddr> {
|
||||
|
||||
if let Ok(addrs) = getifaddrs() {
|
||||
for iface in addrs {
|
||||
if iface.interface_name == name {
|
||||
if let Some(address) = iface.address {
|
||||
if let Some(v4) = address.as_sockaddr_in() {
|
||||
if !want_ipv6 {
|
||||
return Some(IpAddr::V4(v4.ip()));
|
||||
}
|
||||
} else if let Some(v6) = address.as_sockaddr_in6() {
|
||||
if want_ipv6 {
|
||||
return Some(IpAddr::V6(v6.ip().clone()));
|
||||
}
|
||||
if iface.interface_name == name
|
||||
&& let Some(address) = iface.address
|
||||
{
|
||||
if let Some(v4) = address.as_sockaddr_in() {
|
||||
if !want_ipv6 {
|
||||
return Some(IpAddr::V4(v4.ip()));
|
||||
}
|
||||
} else if let Some(v6) = address.as_sockaddr_in6()
|
||||
&& want_ipv6
|
||||
{
|
||||
return Some(IpAddr::V6(v6.ip()));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -157,11 +161,13 @@ pub fn resolve_interface_ip(_name: &str, _want_ipv6: bool) -> Option<IpAddr> {
|
||||
}
|
||||
|
||||
/// Get peer address of a socket
|
||||
#[allow(dead_code)]
|
||||
pub fn get_peer_addr(stream: &TcpStream) -> Option<SocketAddr> {
|
||||
stream.peer_addr().ok()
|
||||
}
|
||||
|
||||
/// Check if address is IPv6
|
||||
#[allow(dead_code)]
|
||||
pub fn is_ipv6(addr: &SocketAddr) -> bool {
|
||||
addr.is_ipv6()
|
||||
}
|
||||
|
||||
@@ -1,15 +1,20 @@
|
||||
//! SOCKS4/5 Client Implementation
|
||||
|
||||
use std::net::{IpAddr, SocketAddr};
|
||||
use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt};
|
||||
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
||||
use tokio::net::TcpStream;
|
||||
use crate::error::{ProxyError, Result};
|
||||
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
pub struct SocksBoundAddr {
|
||||
pub addr: SocketAddr,
|
||||
}
|
||||
|
||||
pub async fn connect_socks4(
|
||||
stream: &mut TcpStream,
|
||||
target: SocketAddr,
|
||||
user_id: Option<&str>,
|
||||
) -> Result<()> {
|
||||
) -> Result<SocksBoundAddr> {
|
||||
let ip = match target.ip() {
|
||||
IpAddr::V4(ip) => ip,
|
||||
IpAddr::V6(_) => return Err(ProxyError::Proxy("SOCKS4 does not support IPv6".to_string())),
|
||||
@@ -27,17 +32,22 @@ pub async fn connect_socks4(
|
||||
buf.extend_from_slice(user);
|
||||
buf.push(0); // NULL
|
||||
|
||||
stream.write_all(&buf).await.map_err(|e| ProxyError::Io(e))?;
|
||||
stream.write_all(&buf).await.map_err(ProxyError::Io)?;
|
||||
|
||||
// Response: VN (1) | CD (1) | DSTPORT (2) | DSTIP (4)
|
||||
let mut resp = [0u8; 8];
|
||||
stream.read_exact(&mut resp).await.map_err(|e| ProxyError::Io(e))?;
|
||||
stream.read_exact(&mut resp).await.map_err(ProxyError::Io)?;
|
||||
|
||||
if resp[1] != 90 {
|
||||
return Err(ProxyError::Proxy(format!("SOCKS4 request rejected: code {}", resp[1])));
|
||||
}
|
||||
|
||||
Ok(())
|
||||
|
||||
let bound_port = u16::from_be_bytes([resp[2], resp[3]]);
|
||||
let bound_ip = IpAddr::from([resp[4], resp[5], resp[6], resp[7]]);
|
||||
|
||||
Ok(SocksBoundAddr {
|
||||
addr: SocketAddr::new(bound_ip, bound_port),
|
||||
})
|
||||
}
|
||||
|
||||
pub async fn connect_socks5(
|
||||
@@ -45,7 +55,7 @@ pub async fn connect_socks5(
|
||||
target: SocketAddr,
|
||||
username: Option<&str>,
|
||||
password: Option<&str>,
|
||||
) -> Result<()> {
|
||||
) -> Result<SocksBoundAddr> {
|
||||
// 1. Auth negotiation
|
||||
// VER (1) | NMETHODS (1) | METHODS (variable)
|
||||
let mut methods = vec![0u8]; // No auth
|
||||
@@ -56,10 +66,10 @@ pub async fn connect_socks5(
|
||||
let mut buf = vec![5u8, methods.len() as u8];
|
||||
buf.extend_from_slice(&methods);
|
||||
|
||||
stream.write_all(&buf).await.map_err(|e| ProxyError::Io(e))?;
|
||||
stream.write_all(&buf).await.map_err(ProxyError::Io)?;
|
||||
|
||||
let mut resp = [0u8; 2];
|
||||
stream.read_exact(&mut resp).await.map_err(|e| ProxyError::Io(e))?;
|
||||
stream.read_exact(&mut resp).await.map_err(ProxyError::Io)?;
|
||||
|
||||
if resp[0] != 5 {
|
||||
return Err(ProxyError::Proxy("Invalid SOCKS5 version".to_string()));
|
||||
@@ -80,10 +90,10 @@ pub async fn connect_socks5(
|
||||
auth_buf.push(p_bytes.len() as u8);
|
||||
auth_buf.extend_from_slice(p_bytes);
|
||||
|
||||
stream.write_all(&auth_buf).await.map_err(|e| ProxyError::Io(e))?;
|
||||
stream.write_all(&auth_buf).await.map_err(ProxyError::Io)?;
|
||||
|
||||
let mut auth_resp = [0u8; 2];
|
||||
stream.read_exact(&mut auth_resp).await.map_err(|e| ProxyError::Io(e))?;
|
||||
stream.read_exact(&mut auth_resp).await.map_err(ProxyError::Io)?;
|
||||
|
||||
if auth_resp[1] != 0 {
|
||||
return Err(ProxyError::Proxy("SOCKS5 authentication failed".to_string()));
|
||||
@@ -112,34 +122,46 @@ pub async fn connect_socks5(
|
||||
|
||||
req.extend_from_slice(&target.port().to_be_bytes());
|
||||
|
||||
stream.write_all(&req).await.map_err(|e| ProxyError::Io(e))?;
|
||||
stream.write_all(&req).await.map_err(ProxyError::Io)?;
|
||||
|
||||
// Response
|
||||
let mut head = [0u8; 4];
|
||||
stream.read_exact(&mut head).await.map_err(|e| ProxyError::Io(e))?;
|
||||
stream.read_exact(&mut head).await.map_err(ProxyError::Io)?;
|
||||
|
||||
if head[1] != 0 {
|
||||
return Err(ProxyError::Proxy(format!("SOCKS5 request failed: code {}", head[1])));
|
||||
}
|
||||
|
||||
// Skip address part of response
|
||||
match head[3] {
|
||||
// Parse bound address from response.
|
||||
let bound_addr = match head[3] {
|
||||
1 => { // IPv4
|
||||
let mut addr = [0u8; 4 + 2];
|
||||
stream.read_exact(&mut addr).await.map_err(|e| ProxyError::Io(e))?;
|
||||
stream.read_exact(&mut addr).await.map_err(ProxyError::Io)?;
|
||||
let ip = IpAddr::from([addr[0], addr[1], addr[2], addr[3]]);
|
||||
let port = u16::from_be_bytes([addr[4], addr[5]]);
|
||||
SocketAddr::new(ip, port)
|
||||
},
|
||||
3 => { // Domain
|
||||
let mut len = [0u8; 1];
|
||||
stream.read_exact(&mut len).await.map_err(|e| ProxyError::Io(e))?;
|
||||
stream.read_exact(&mut len).await.map_err(ProxyError::Io)?;
|
||||
let mut addr = vec![0u8; len[0] as usize + 2];
|
||||
stream.read_exact(&mut addr).await.map_err(|e| ProxyError::Io(e))?;
|
||||
stream.read_exact(&mut addr).await.map_err(ProxyError::Io)?;
|
||||
// Domain-bound response is not useful for KDF IP material.
|
||||
let port_pos = addr.len().saturating_sub(2);
|
||||
let port = u16::from_be_bytes([addr[port_pos], addr[port_pos + 1]]);
|
||||
SocketAddr::new(IpAddr::from([0, 0, 0, 0]), port)
|
||||
},
|
||||
4 => { // IPv6
|
||||
let mut addr = [0u8; 16 + 2];
|
||||
stream.read_exact(&mut addr).await.map_err(|e| ProxyError::Io(e))?;
|
||||
stream.read_exact(&mut addr).await.map_err(ProxyError::Io)?;
|
||||
let ip = IpAddr::from(<[u8; 16]>::try_from(&addr[..16]).map_err(|_| {
|
||||
ProxyError::Proxy("Invalid SOCKS5 IPv6 bound address".to_string())
|
||||
})?);
|
||||
let port = u16::from_be_bytes([addr[16], addr[17]]);
|
||||
SocketAddr::new(ip, port)
|
||||
},
|
||||
_ => return Err(ProxyError::Proxy("Invalid address type in SOCKS5 response".to_string())),
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
};
|
||||
|
||||
Ok(SocksBoundAddr { addr: bound_addr })
|
||||
}
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
//! Upstream Management with per-DC latency-weighted selection
|
||||
//!
|
||||
//!
|
||||
//! IPv6/IPv4 connectivity checks with configurable preference.
|
||||
|
||||
#![allow(deprecated)]
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::net::{SocketAddr, IpAddr};
|
||||
use std::sync::Arc;
|
||||
@@ -15,6 +17,7 @@ use tracing::{debug, warn, info, trace};
|
||||
|
||||
use crate::config::{UpstreamConfig, UpstreamType};
|
||||
use crate::error::{Result, ProxyError};
|
||||
use crate::network::dns_overrides::{resolve_socket_addr, split_host_port};
|
||||
use crate::protocol::constants::{TG_DATACENTERS_V4, TG_DATACENTERS_V6, TG_DATACENTER_PORT};
|
||||
use crate::transport::socket::{create_outgoing_socket_bound, resolve_interface_ip};
|
||||
use crate::transport::socks::{connect_socks4, connect_socks5};
|
||||
@@ -55,9 +58,10 @@ impl LatencyEma {
|
||||
// ============= Per-DC IP Preference Tracking =============
|
||||
|
||||
/// Tracks which IP version works for each DC
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
|
||||
pub enum IpPreference {
|
||||
/// Not yet tested
|
||||
#[default]
|
||||
Unknown,
|
||||
/// IPv6 works
|
||||
PreferV6,
|
||||
@@ -69,12 +73,6 @@ pub enum IpPreference {
|
||||
Unavailable,
|
||||
}
|
||||
|
||||
impl Default for IpPreference {
|
||||
fn default() -> Self {
|
||||
Self::Unknown
|
||||
}
|
||||
}
|
||||
|
||||
// ============= Upstream State =============
|
||||
|
||||
#[derive(Debug)]
|
||||
@@ -110,7 +108,7 @@ impl UpstreamState {
|
||||
if abs_dc == 0 {
|
||||
return None;
|
||||
}
|
||||
if abs_dc >= 1 && abs_dc <= NUM_DCS {
|
||||
if (1..=NUM_DCS).contains(&abs_dc) {
|
||||
Some(abs_dc - 1)
|
||||
} else {
|
||||
// Unknown DC → default cluster (DC 2, index 1)
|
||||
@@ -120,10 +118,10 @@ impl UpstreamState {
|
||||
|
||||
/// Get latency for a specific DC, falling back to average across all known DCs
|
||||
fn effective_latency(&self, dc_idx: Option<i16>) -> Option<f64> {
|
||||
if let Some(di) = dc_idx.and_then(Self::dc_array_idx) {
|
||||
if let Some(ms) = self.dc_latency[di].get() {
|
||||
return Some(ms);
|
||||
}
|
||||
if let Some(di) = dc_idx.and_then(Self::dc_array_idx)
|
||||
&& let Some(ms) = self.dc_latency[di].get()
|
||||
{
|
||||
return Some(ms);
|
||||
}
|
||||
|
||||
let (sum, count) = self.dc_latency.iter()
|
||||
@@ -153,15 +151,39 @@ pub struct StartupPingResult {
|
||||
pub both_available: bool,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum UpstreamRouteKind {
|
||||
Direct,
|
||||
Socks4,
|
||||
Socks5,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub struct UpstreamEgressInfo {
|
||||
pub route_kind: UpstreamRouteKind,
|
||||
pub local_addr: Option<SocketAddr>,
|
||||
pub direct_bind_ip: Option<IpAddr>,
|
||||
pub socks_bound_addr: Option<SocketAddr>,
|
||||
pub socks_proxy_addr: Option<SocketAddr>,
|
||||
}
|
||||
|
||||
// ============= Upstream Manager =============
|
||||
|
||||
#[derive(Clone)]
|
||||
pub struct UpstreamManager {
|
||||
upstreams: Arc<RwLock<Vec<UpstreamState>>>,
|
||||
connect_retry_attempts: u32,
|
||||
connect_retry_backoff: Duration,
|
||||
unhealthy_fail_threshold: u32,
|
||||
}
|
||||
|
||||
impl UpstreamManager {
|
||||
pub fn new(configs: Vec<UpstreamConfig>) -> Self {
|
||||
pub fn new(
|
||||
configs: Vec<UpstreamConfig>,
|
||||
connect_retry_attempts: u32,
|
||||
connect_retry_backoff_ms: u64,
|
||||
unhealthy_fail_threshold: u32,
|
||||
) -> Self {
|
||||
let states = configs.into_iter()
|
||||
.filter(|c| c.enabled)
|
||||
.map(UpstreamState::new)
|
||||
@@ -169,24 +191,88 @@ impl UpstreamManager {
|
||||
|
||||
Self {
|
||||
upstreams: Arc::new(RwLock::new(states)),
|
||||
connect_retry_attempts: connect_retry_attempts.max(1),
|
||||
connect_retry_backoff: Duration::from_millis(connect_retry_backoff_ms),
|
||||
unhealthy_fail_threshold: unhealthy_fail_threshold.max(1),
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
fn resolve_interface_addrs(name: &str, want_ipv6: bool) -> Vec<IpAddr> {
|
||||
use nix::ifaddrs::getifaddrs;
|
||||
|
||||
let mut out = Vec::new();
|
||||
if let Ok(addrs) = getifaddrs() {
|
||||
for iface in addrs {
|
||||
if iface.interface_name != name {
|
||||
continue;
|
||||
}
|
||||
if let Some(address) = iface.address {
|
||||
if let Some(v4) = address.as_sockaddr_in() {
|
||||
if !want_ipv6 {
|
||||
out.push(IpAddr::V4(v4.ip()));
|
||||
}
|
||||
} else if let Some(v6) = address.as_sockaddr_in6()
|
||||
&& want_ipv6
|
||||
{
|
||||
out.push(IpAddr::V6(v6.ip()));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
out.sort_unstable();
|
||||
out.dedup();
|
||||
out
|
||||
}
|
||||
|
||||
fn resolve_bind_address(
|
||||
interface: &Option<String>,
|
||||
bind_addresses: &Option<Vec<String>>,
|
||||
target: SocketAddr,
|
||||
rr: Option<&AtomicUsize>,
|
||||
validate_ip_on_interface: bool,
|
||||
) -> Option<IpAddr> {
|
||||
let want_ipv6 = target.is_ipv6();
|
||||
|
||||
if let Some(addrs) = bind_addresses {
|
||||
let candidates: Vec<IpAddr> = addrs
|
||||
let mut candidates: Vec<IpAddr> = addrs
|
||||
.iter()
|
||||
.filter_map(|s| s.parse::<IpAddr>().ok())
|
||||
.filter(|ip| ip.is_ipv6() == want_ipv6)
|
||||
.collect();
|
||||
|
||||
// Explicit bind IP has strict priority over interface auto-selection.
|
||||
if validate_ip_on_interface
|
||||
&& let Some(iface) = interface
|
||||
&& iface.parse::<IpAddr>().is_err()
|
||||
{
|
||||
#[cfg(unix)]
|
||||
{
|
||||
let iface_addrs = Self::resolve_interface_addrs(iface, want_ipv6);
|
||||
if !iface_addrs.is_empty() {
|
||||
candidates.retain(|ip| {
|
||||
let ok = iface_addrs.contains(ip);
|
||||
if !ok {
|
||||
warn!(
|
||||
interface = %iface,
|
||||
bind_ip = %ip,
|
||||
target = %target,
|
||||
"Configured bind address is not assigned to interface"
|
||||
);
|
||||
}
|
||||
ok
|
||||
});
|
||||
} else if !candidates.is_empty() {
|
||||
warn!(
|
||||
interface = %iface,
|
||||
target = %target,
|
||||
"Configured interface has no addresses for target family; falling back to direct connect without bind"
|
||||
);
|
||||
candidates.clear();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if !candidates.is_empty() {
|
||||
if let Some(counter) = rr {
|
||||
let idx = counter.fetch_add(1, Ordering::Relaxed) % candidates.len();
|
||||
@@ -194,6 +280,19 @@ impl UpstreamManager {
|
||||
}
|
||||
return candidates.first().copied();
|
||||
}
|
||||
|
||||
if validate_ip_on_interface
|
||||
&& interface
|
||||
.as_ref()
|
||||
.is_some_and(|iface| iface.parse::<IpAddr>().is_err())
|
||||
{
|
||||
warn!(
|
||||
interface = interface.as_deref().unwrap_or(""),
|
||||
target = %target,
|
||||
"No valid bind_addresses left for interface; falling back to direct connect without bind"
|
||||
);
|
||||
return None;
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(iface) = interface {
|
||||
@@ -212,6 +311,31 @@ impl UpstreamManager {
|
||||
None
|
||||
}
|
||||
|
||||
async fn connect_hostname_with_dns_override(
|
||||
address: &str,
|
||||
connect_timeout: Duration,
|
||||
) -> Result<TcpStream> {
|
||||
if let Some((host, port)) = split_host_port(address)
|
||||
&& let Some(addr) = resolve_socket_addr(&host, port)
|
||||
{
|
||||
return match tokio::time::timeout(connect_timeout, TcpStream::connect(addr)).await {
|
||||
Ok(Ok(stream)) => Ok(stream),
|
||||
Ok(Err(e)) => Err(ProxyError::Io(e)),
|
||||
Err(_) => Err(ProxyError::ConnectionTimeout {
|
||||
addr: addr.to_string(),
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
match tokio::time::timeout(connect_timeout, TcpStream::connect(address)).await {
|
||||
Ok(Ok(stream)) => Ok(stream),
|
||||
Ok(Err(e)) => Err(ProxyError::Io(e)),
|
||||
Err(_) => Err(ProxyError::ConnectionTimeout {
|
||||
addr: address.to_string(),
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
/// Select upstream using latency-weighted random selection.
|
||||
async fn select_upstream(&self, dc_idx: Option<i16>, scope: Option<&str>) -> Option<usize> {
|
||||
let upstreams = self.upstreams.read().await;
|
||||
@@ -293,6 +417,17 @@ impl UpstreamManager {
|
||||
|
||||
/// Connect to target through a selected upstream.
|
||||
pub async fn connect(&self, target: SocketAddr, dc_idx: Option<i16>, scope: Option<&str>) -> Result<TcpStream> {
|
||||
let (stream, _) = self.connect_with_details(target, dc_idx, scope).await?;
|
||||
Ok(stream)
|
||||
}
|
||||
|
||||
/// Connect to target through a selected upstream and return egress details.
|
||||
pub async fn connect_with_details(
|
||||
&self,
|
||||
target: SocketAddr,
|
||||
dc_idx: Option<i16>,
|
||||
scope: Option<&str>,
|
||||
) -> Result<(TcpStream, UpstreamEgressInfo)> {
|
||||
let idx = self.select_upstream(dc_idx, scope).await
|
||||
.ok_or_else(|| ProxyError::Config("No upstreams available".to_string()))?;
|
||||
|
||||
@@ -306,43 +441,83 @@ impl UpstreamManager {
|
||||
upstream.selected_scope = s.to_string();
|
||||
}
|
||||
|
||||
let start = Instant::now();
|
||||
|
||||
let bind_rr = {
|
||||
let guard = self.upstreams.read().await;
|
||||
guard.get(idx).map(|u| u.bind_rr.clone())
|
||||
};
|
||||
|
||||
match self.connect_via_upstream(&upstream, target, bind_rr).await {
|
||||
Ok(stream) => {
|
||||
let rtt_ms = start.elapsed().as_secs_f64() * 1000.0;
|
||||
let mut guard = self.upstreams.write().await;
|
||||
if let Some(u) = guard.get_mut(idx) {
|
||||
if !u.healthy {
|
||||
debug!(rtt_ms = format!("{:.1}", rtt_ms), "Upstream recovered");
|
||||
}
|
||||
u.healthy = true;
|
||||
u.fails = 0;
|
||||
let mut last_error: Option<ProxyError> = None;
|
||||
for attempt in 1..=self.connect_retry_attempts {
|
||||
let start = Instant::now();
|
||||
match self
|
||||
.connect_via_upstream(&upstream, target, bind_rr.clone())
|
||||
.await
|
||||
{
|
||||
Ok((stream, egress)) => {
|
||||
let rtt_ms = start.elapsed().as_secs_f64() * 1000.0;
|
||||
let mut guard = self.upstreams.write().await;
|
||||
if let Some(u) = guard.get_mut(idx) {
|
||||
if !u.healthy {
|
||||
debug!(rtt_ms = format!("{:.1}", rtt_ms), "Upstream recovered");
|
||||
}
|
||||
if attempt > 1 {
|
||||
debug!(
|
||||
attempt,
|
||||
attempts = self.connect_retry_attempts,
|
||||
rtt_ms = format!("{:.1}", rtt_ms),
|
||||
"Upstream connect recovered after retry"
|
||||
);
|
||||
}
|
||||
u.healthy = true;
|
||||
u.fails = 0;
|
||||
|
||||
if let Some(di) = dc_idx.and_then(UpstreamState::dc_array_idx) {
|
||||
u.dc_latency[di].update(rtt_ms);
|
||||
if let Some(di) = dc_idx.and_then(UpstreamState::dc_array_idx) {
|
||||
u.dc_latency[di].update(rtt_ms);
|
||||
}
|
||||
}
|
||||
return Ok((stream, egress));
|
||||
}
|
||||
Ok(stream)
|
||||
},
|
||||
Err(e) => {
|
||||
let mut guard = self.upstreams.write().await;
|
||||
if let Some(u) = guard.get_mut(idx) {
|
||||
u.fails += 1;
|
||||
warn!(fails = u.fails, "Upstream failed: {}", e);
|
||||
if u.fails > 3 {
|
||||
u.healthy = false;
|
||||
warn!("Upstream marked unhealthy");
|
||||
Err(e) => {
|
||||
if attempt < self.connect_retry_attempts {
|
||||
debug!(
|
||||
attempt,
|
||||
attempts = self.connect_retry_attempts,
|
||||
target = %target,
|
||||
error = %e,
|
||||
"Upstream connect attempt failed, retrying"
|
||||
);
|
||||
if !self.connect_retry_backoff.is_zero() {
|
||||
tokio::time::sleep(self.connect_retry_backoff).await;
|
||||
}
|
||||
}
|
||||
last_error = Some(e);
|
||||
}
|
||||
Err(e)
|
||||
}
|
||||
}
|
||||
|
||||
let error = last_error.unwrap_or_else(|| {
|
||||
ProxyError::Config("Upstream connect attempts exhausted".to_string())
|
||||
});
|
||||
|
||||
let mut guard = self.upstreams.write().await;
|
||||
if let Some(u) = guard.get_mut(idx) {
|
||||
u.fails += 1;
|
||||
warn!(
|
||||
fails = u.fails,
|
||||
attempts = self.connect_retry_attempts,
|
||||
"Upstream failed after retries: {}",
|
||||
error
|
||||
);
|
||||
if u.fails >= self.unhealthy_fail_threshold {
|
||||
u.healthy = false;
|
||||
warn!(
|
||||
fails = u.fails,
|
||||
threshold = self.unhealthy_fail_threshold,
|
||||
"Upstream marked unhealthy"
|
||||
);
|
||||
}
|
||||
}
|
||||
Err(error)
|
||||
}
|
||||
|
||||
async fn connect_via_upstream(
|
||||
@@ -350,7 +525,7 @@ impl UpstreamManager {
|
||||
config: &UpstreamConfig,
|
||||
target: SocketAddr,
|
||||
bind_rr: Option<Arc<AtomicUsize>>,
|
||||
) -> Result<TcpStream> {
|
||||
) -> Result<(TcpStream, UpstreamEgressInfo)> {
|
||||
match &config.upstream_type {
|
||||
UpstreamType::Direct { interface, bind_addresses } => {
|
||||
let bind_ip = Self::resolve_bind_address(
|
||||
@@ -358,6 +533,7 @@ impl UpstreamManager {
|
||||
bind_addresses,
|
||||
target,
|
||||
bind_rr.as_deref(),
|
||||
true,
|
||||
);
|
||||
|
||||
let socket = create_outgoing_socket_bound(target, bind_ip)?;
|
||||
@@ -391,7 +567,17 @@ impl UpstreamManager {
|
||||
return Err(ProxyError::Io(e));
|
||||
}
|
||||
|
||||
Ok(stream)
|
||||
let local_addr = stream.local_addr().ok();
|
||||
Ok((
|
||||
stream,
|
||||
UpstreamEgressInfo {
|
||||
route_kind: UpstreamRouteKind::Direct,
|
||||
local_addr,
|
||||
direct_bind_ip: bind_ip,
|
||||
socks_bound_addr: None,
|
||||
socks_proxy_addr: None,
|
||||
},
|
||||
))
|
||||
},
|
||||
UpstreamType::Socks4 { address, interface, user_id } => {
|
||||
let connect_timeout = Duration::from_secs(DIRECT_CONNECT_TIMEOUT_SECS);
|
||||
@@ -403,6 +589,7 @@ impl UpstreamManager {
|
||||
&None,
|
||||
proxy_addr,
|
||||
bind_rr.as_deref(),
|
||||
false,
|
||||
);
|
||||
|
||||
let socket = create_outgoing_socket_bound(proxy_addr, bind_ip)?;
|
||||
@@ -436,15 +623,7 @@ impl UpstreamManager {
|
||||
if interface.is_some() {
|
||||
warn!("SOCKS4 interface binding is not supported for hostname addresses, ignoring");
|
||||
}
|
||||
match tokio::time::timeout(connect_timeout, TcpStream::connect(address)).await {
|
||||
Ok(Ok(stream)) => stream,
|
||||
Ok(Err(e)) => return Err(ProxyError::Io(e)),
|
||||
Err(_) => {
|
||||
return Err(ProxyError::ConnectionTimeout {
|
||||
addr: address.clone(),
|
||||
});
|
||||
}
|
||||
}
|
||||
Self::connect_hostname_with_dns_override(address, connect_timeout).await?
|
||||
};
|
||||
|
||||
// replace socks user_id with config.selected_scope, if set
|
||||
@@ -452,16 +631,32 @@ impl UpstreamManager {
|
||||
.filter(|s| !s.is_empty());
|
||||
let _user_id: Option<&str> = scope.or(user_id.as_deref());
|
||||
|
||||
match tokio::time::timeout(connect_timeout, connect_socks4(&mut stream, target, _user_id)).await {
|
||||
Ok(Ok(())) => {}
|
||||
let bound = match tokio::time::timeout(
|
||||
connect_timeout,
|
||||
connect_socks4(&mut stream, target, _user_id),
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(Ok(bound)) => bound,
|
||||
Ok(Err(e)) => return Err(e),
|
||||
Err(_) => {
|
||||
return Err(ProxyError::ConnectionTimeout {
|
||||
addr: target.to_string(),
|
||||
});
|
||||
}
|
||||
}
|
||||
Ok(stream)
|
||||
};
|
||||
let local_addr = stream.local_addr().ok();
|
||||
let socks_proxy_addr = stream.peer_addr().ok();
|
||||
Ok((
|
||||
stream,
|
||||
UpstreamEgressInfo {
|
||||
route_kind: UpstreamRouteKind::Socks4,
|
||||
local_addr,
|
||||
direct_bind_ip: None,
|
||||
socks_bound_addr: Some(bound.addr),
|
||||
socks_proxy_addr,
|
||||
},
|
||||
))
|
||||
},
|
||||
UpstreamType::Socks5 { address, interface, username, password } => {
|
||||
let connect_timeout = Duration::from_secs(DIRECT_CONNECT_TIMEOUT_SECS);
|
||||
@@ -473,6 +668,7 @@ impl UpstreamManager {
|
||||
&None,
|
||||
proxy_addr,
|
||||
bind_rr.as_deref(),
|
||||
false,
|
||||
);
|
||||
|
||||
let socket = create_outgoing_socket_bound(proxy_addr, bind_ip)?;
|
||||
@@ -506,15 +702,7 @@ impl UpstreamManager {
|
||||
if interface.is_some() {
|
||||
warn!("SOCKS5 interface binding is not supported for hostname addresses, ignoring");
|
||||
}
|
||||
match tokio::time::timeout(connect_timeout, TcpStream::connect(address)).await {
|
||||
Ok(Ok(stream)) => stream,
|
||||
Ok(Err(e)) => return Err(ProxyError::Io(e)),
|
||||
Err(_) => {
|
||||
return Err(ProxyError::ConnectionTimeout {
|
||||
addr: address.clone(),
|
||||
});
|
||||
}
|
||||
}
|
||||
Self::connect_hostname_with_dns_override(address, connect_timeout).await?
|
||||
};
|
||||
|
||||
debug!(config = ?config, "Socks5 connection");
|
||||
@@ -524,21 +712,32 @@ impl UpstreamManager {
|
||||
let _username: Option<&str> = scope.or(username.as_deref());
|
||||
let _password: Option<&str> = scope.or(password.as_deref());
|
||||
|
||||
match tokio::time::timeout(
|
||||
let bound = match tokio::time::timeout(
|
||||
connect_timeout,
|
||||
connect_socks5(&mut stream, target, _username, _password),
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(Ok(())) => {}
|
||||
Ok(Ok(bound)) => bound,
|
||||
Ok(Err(e)) => return Err(e),
|
||||
Err(_) => {
|
||||
return Err(ProxyError::ConnectionTimeout {
|
||||
addr: target.to_string(),
|
||||
});
|
||||
}
|
||||
}
|
||||
Ok(stream)
|
||||
};
|
||||
let local_addr = stream.local_addr().ok();
|
||||
let socks_proxy_addr = stream.peer_addr().ok();
|
||||
Ok((
|
||||
stream,
|
||||
UpstreamEgressInfo {
|
||||
route_kind: UpstreamRouteKind::Socks5,
|
||||
local_addr,
|
||||
direct_bind_ip: None,
|
||||
socks_bound_addr: Some(bound.addr),
|
||||
socks_proxy_addr,
|
||||
},
|
||||
))
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -549,7 +748,7 @@ impl UpstreamManager {
|
||||
/// Tests BOTH IPv6 and IPv4, returns separate results for each.
|
||||
pub async fn ping_all_dcs(
|
||||
&self,
|
||||
prefer_ipv6: bool,
|
||||
_prefer_ipv6: bool,
|
||||
dc_overrides: &HashMap<String, Vec<String>>,
|
||||
ipv4_enabled: bool,
|
||||
ipv6_enabled: bool,
|
||||
@@ -565,8 +764,22 @@ impl UpstreamManager {
|
||||
|
||||
for (upstream_idx, upstream_config, bind_rr) in &upstreams {
|
||||
let upstream_name = match &upstream_config.upstream_type {
|
||||
UpstreamType::Direct { interface, .. } => {
|
||||
format!("direct{}", interface.as_ref().map(|i| format!(" ({})", i)).unwrap_or_default())
|
||||
UpstreamType::Direct {
|
||||
interface,
|
||||
bind_addresses,
|
||||
} => {
|
||||
let mut direct_parts = Vec::new();
|
||||
if let Some(dev) = interface.as_deref().filter(|v| !v.is_empty()) {
|
||||
direct_parts.push(format!("dev={dev}"));
|
||||
}
|
||||
if let Some(src) = bind_addresses.as_ref().filter(|v| !v.is_empty()) {
|
||||
direct_parts.push(format!("src={}", src.join(",")));
|
||||
}
|
||||
if direct_parts.is_empty() {
|
||||
"direct".to_string()
|
||||
} else {
|
||||
format!("direct {}", direct_parts.join(" "))
|
||||
}
|
||||
}
|
||||
UpstreamType::Socks4 { address, .. } => format!("socks4://{}", address),
|
||||
UpstreamType::Socks5 { address, .. } => format!("socks5://{}", address),
|
||||
@@ -580,7 +793,7 @@ impl UpstreamManager {
|
||||
|
||||
let result = tokio::time::timeout(
|
||||
Duration::from_secs(DC_PING_TIMEOUT_SECS),
|
||||
self.ping_single_dc(&upstream_config, Some(bind_rr.clone()), addr_v6)
|
||||
self.ping_single_dc(upstream_config, Some(bind_rr.clone()), addr_v6)
|
||||
).await;
|
||||
|
||||
let ping_result = match result {
|
||||
@@ -631,7 +844,7 @@ impl UpstreamManager {
|
||||
|
||||
let result = tokio::time::timeout(
|
||||
Duration::from_secs(DC_PING_TIMEOUT_SECS),
|
||||
self.ping_single_dc(&upstream_config, Some(bind_rr.clone()), addr_v4)
|
||||
self.ping_single_dc(upstream_config, Some(bind_rr.clone()), addr_v4)
|
||||
).await;
|
||||
|
||||
let ping_result = match result {
|
||||
@@ -694,7 +907,7 @@ impl UpstreamManager {
|
||||
}
|
||||
let result = tokio::time::timeout(
|
||||
Duration::from_secs(DC_PING_TIMEOUT_SECS),
|
||||
self.ping_single_dc(&upstream_config, Some(bind_rr.clone()), addr)
|
||||
self.ping_single_dc(upstream_config, Some(bind_rr.clone()), addr)
|
||||
).await;
|
||||
|
||||
let ping_result = match result {
|
||||
@@ -770,7 +983,7 @@ impl UpstreamManager {
|
||||
target: SocketAddr,
|
||||
) -> Result<f64> {
|
||||
let start = Instant::now();
|
||||
let _stream = self.connect_via_upstream(config, target, bind_rr).await?;
|
||||
let _ = self.connect_via_upstream(config, target, bind_rr).await?;
|
||||
Ok(start.elapsed().as_secs_f64() * 1000.0)
|
||||
}
|
||||
|
||||
@@ -873,18 +1086,26 @@ impl UpstreamManager {
|
||||
u.fails += 1;
|
||||
debug!(dc = dc_zero_idx + 1, fails = u.fails,
|
||||
"Health check failed (both): {}", e);
|
||||
if u.fails > 3 {
|
||||
if u.fails >= self.unhealthy_fail_threshold {
|
||||
u.healthy = false;
|
||||
warn!("Upstream unhealthy (fails)");
|
||||
warn!(
|
||||
fails = u.fails,
|
||||
threshold = self.unhealthy_fail_threshold,
|
||||
"Upstream unhealthy (fails)"
|
||||
);
|
||||
}
|
||||
}
|
||||
Err(_) => {
|
||||
u.fails += 1;
|
||||
debug!(dc = dc_zero_idx + 1, fails = u.fails,
|
||||
"Health check timeout (both)");
|
||||
if u.fails > 3 {
|
||||
if u.fails >= self.unhealthy_fail_threshold {
|
||||
u.healthy = false;
|
||||
warn!("Upstream unhealthy (timeout)");
|
||||
warn!(
|
||||
fails = u.fails,
|
||||
threshold = self.unhealthy_fail_threshold,
|
||||
"Upstream unhealthy (timeout)"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -895,9 +1116,13 @@ impl UpstreamManager {
|
||||
let mut guard = self.upstreams.write().await;
|
||||
let u = &mut guard[i];
|
||||
u.fails += 1;
|
||||
if u.fails > 3 {
|
||||
if u.fails >= self.unhealthy_fail_threshold {
|
||||
u.healthy = false;
|
||||
warn!("Upstream unhealthy (no fallback family)");
|
||||
warn!(
|
||||
fails = u.fails,
|
||||
threshold = self.unhealthy_fail_threshold,
|
||||
"Upstream unhealthy (no fallback family)"
|
||||
);
|
||||
}
|
||||
u.last_check = std::time::Instant::now();
|
||||
}
|
||||
@@ -907,6 +1132,7 @@ impl UpstreamManager {
|
||||
}
|
||||
|
||||
/// Get the preferred IP for a DC (for use by other components)
|
||||
#[allow(dead_code)]
|
||||
pub async fn get_dc_ip_preference(&self, dc_idx: i16) -> Option<IpPreference> {
|
||||
let guard = self.upstreams.read().await;
|
||||
if guard.is_empty() {
|
||||
@@ -918,6 +1144,7 @@ impl UpstreamManager {
|
||||
}
|
||||
|
||||
/// Get preferred DC address based on config preference
|
||||
#[allow(dead_code)]
|
||||
pub async fn get_dc_addr(&self, dc_idx: i16, prefer_ipv6: bool) -> Option<SocketAddr> {
|
||||
let arr_idx = UpstreamState::dc_array_idx(dc_idx)?;
|
||||
|
||||
|
||||
@@ -1,22 +1,24 @@
|
||||
//! IP Addr Detect
|
||||
|
||||
use std::net::{IpAddr, SocketAddr, UdpSocket};
|
||||
use std::net::{IpAddr, UdpSocket};
|
||||
use std::time::Duration;
|
||||
use tracing::{debug, warn};
|
||||
|
||||
/// Detected IP addresses
|
||||
#[derive(Debug, Clone, Default)]
|
||||
#[allow(dead_code)]
|
||||
pub struct IpInfo {
|
||||
pub ipv4: Option<IpAddr>,
|
||||
pub ipv6: Option<IpAddr>,
|
||||
}
|
||||
|
||||
#[allow(dead_code)]
|
||||
impl IpInfo {
|
||||
/// Check if any IP is detected
|
||||
pub fn has_any(&self) -> bool {
|
||||
self.ipv4.is_some() || self.ipv6.is_some()
|
||||
}
|
||||
|
||||
|
||||
/// Get preferred IP (IPv6 if available and preferred)
|
||||
pub fn preferred(&self, prefer_ipv6: bool) -> Option<IpAddr> {
|
||||
if prefer_ipv6 {
|
||||
@@ -28,12 +30,14 @@ impl IpInfo {
|
||||
}
|
||||
|
||||
/// URLs for IP detection
|
||||
#[allow(dead_code)]
|
||||
const IPV4_URLS: &[&str] = &[
|
||||
"http://v4.ident.me/",
|
||||
"http://ipv4.icanhazip.com/",
|
||||
"http://api.ipify.org/",
|
||||
];
|
||||
|
||||
#[allow(dead_code)]
|
||||
const IPV6_URLS: &[&str] = &[
|
||||
"http://v6.ident.me/",
|
||||
"http://ipv6.icanhazip.com/",
|
||||
@@ -42,12 +46,14 @@ const IPV6_URLS: &[&str] = &[
|
||||
|
||||
/// Detect local IP address by connecting to a public DNS
|
||||
/// This does not actually send any packets
|
||||
#[allow(dead_code)]
|
||||
fn get_local_ip(target: &str) -> Option<IpAddr> {
|
||||
let socket = UdpSocket::bind("0.0.0.0:0").ok()?;
|
||||
socket.connect(target).ok()?;
|
||||
socket.local_addr().ok().map(|addr| addr.ip())
|
||||
}
|
||||
|
||||
#[allow(dead_code)]
|
||||
fn get_local_ipv6(target: &str) -> Option<IpAddr> {
|
||||
let socket = UdpSocket::bind("[::]:0").ok()?;
|
||||
socket.connect(target).ok()?;
|
||||
@@ -55,59 +61,62 @@ fn get_local_ipv6(target: &str) -> Option<IpAddr> {
|
||||
}
|
||||
|
||||
/// Detect public IP addresses
|
||||
#[allow(dead_code)]
|
||||
pub async fn detect_ip() -> IpInfo {
|
||||
let mut info = IpInfo::default();
|
||||
|
||||
// Try to get local interface IP first (default gateway interface)
|
||||
// We connect to Google DNS to find out which interface is used for routing
|
||||
if let Some(ip) = get_local_ip("8.8.8.8:80") {
|
||||
if ip.is_ipv4() && !ip.is_loopback() {
|
||||
info.ipv4 = Some(ip);
|
||||
debug!(ip = %ip, "Detected local IPv4 address via routing");
|
||||
}
|
||||
if let Some(ip) = get_local_ip("8.8.8.8:80")
|
||||
&& ip.is_ipv4()
|
||||
&& !ip.is_loopback()
|
||||
{
|
||||
info.ipv4 = Some(ip);
|
||||
debug!(ip = %ip, "Detected local IPv4 address via routing");
|
||||
}
|
||||
|
||||
if let Some(ip) = get_local_ipv6("[2001:4860:4860::8888]:80") {
|
||||
if ip.is_ipv6() && !ip.is_loopback() {
|
||||
info.ipv6 = Some(ip);
|
||||
debug!(ip = %ip, "Detected local IPv6 address via routing");
|
||||
}
|
||||
if let Some(ip) = get_local_ipv6("[2001:4860:4860::8888]:80")
|
||||
&& ip.is_ipv6()
|
||||
&& !ip.is_loopback()
|
||||
{
|
||||
info.ipv6 = Some(ip);
|
||||
debug!(ip = %ip, "Detected local IPv6 address via routing");
|
||||
}
|
||||
|
||||
// If local detection failed or returned private IP (and we want public),
|
||||
|
||||
// If local detection failed or returned private IP (and we want public),
|
||||
// or just as a fallback/verification, we might want to check external services.
|
||||
// However, the requirement is: "if IP for listening is not set... it should be IP from interface...
|
||||
// However, the requirement is: "if IP for listening is not set... it should be IP from interface...
|
||||
// if impossible - request external resources".
|
||||
|
||||
|
||||
// So if we found a local IP, we might be good. But often servers are behind NAT.
|
||||
// If the local IP is private, we probably want the public IP for the tg:// link.
|
||||
// Let's check if the detected IPs are private.
|
||||
|
||||
let need_external_v4 = info.ipv4.map_or(true, |ip| is_private_ip(ip));
|
||||
let need_external_v6 = info.ipv6.map_or(true, |ip| is_private_ip(ip));
|
||||
|
||||
let need_external_v4 = info.ipv4.is_none_or(is_private_ip);
|
||||
let need_external_v6 = info.ipv6.is_none_or(is_private_ip);
|
||||
|
||||
if need_external_v4 {
|
||||
debug!("Local IPv4 is private or missing, checking external services...");
|
||||
for url in IPV4_URLS {
|
||||
if let Some(ip) = fetch_ip(url).await {
|
||||
if ip.is_ipv4() {
|
||||
info.ipv4 = Some(ip);
|
||||
debug!(ip = %ip, "Detected public IPv4 address");
|
||||
break;
|
||||
}
|
||||
if let Some(ip) = fetch_ip(url).await
|
||||
&& ip.is_ipv4()
|
||||
{
|
||||
info.ipv4 = Some(ip);
|
||||
debug!(ip = %ip, "Detected public IPv4 address");
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
if need_external_v6 {
|
||||
debug!("Local IPv6 is private or missing, checking external services...");
|
||||
for url in IPV6_URLS {
|
||||
if let Some(ip) = fetch_ip(url).await {
|
||||
if ip.is_ipv6() {
|
||||
info.ipv6 = Some(ip);
|
||||
debug!(ip = %ip, "Detected public IPv6 address");
|
||||
break;
|
||||
}
|
||||
if let Some(ip) = fetch_ip(url).await
|
||||
&& ip.is_ipv6()
|
||||
{
|
||||
info.ipv6 = Some(ip);
|
||||
debug!(ip = %ip, "Detected public IPv6 address");
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -119,6 +128,7 @@ pub async fn detect_ip() -> IpInfo {
|
||||
info
|
||||
}
|
||||
|
||||
#[allow(dead_code)]
|
||||
fn is_private_ip(ip: IpAddr) -> bool {
|
||||
match ip {
|
||||
IpAddr::V4(ipv4) => {
|
||||
@@ -131,19 +141,21 @@ fn is_private_ip(ip: IpAddr) -> bool {
|
||||
}
|
||||
|
||||
/// Fetch IP from URL
|
||||
#[allow(dead_code)]
|
||||
async fn fetch_ip(url: &str) -> Option<IpAddr> {
|
||||
let client = reqwest::Client::builder()
|
||||
.timeout(Duration::from_secs(5))
|
||||
.build()
|
||||
.ok()?;
|
||||
|
||||
|
||||
let response = client.get(url).send().await.ok()?;
|
||||
let text = response.text().await.ok()?;
|
||||
|
||||
|
||||
text.trim().parse().ok()
|
||||
}
|
||||
|
||||
/// Synchronous IP detection (for startup)
|
||||
#[allow(dead_code)]
|
||||
pub fn detect_ip_sync() -> IpInfo {
|
||||
tokio::runtime::Handle::current().block_on(detect_ip())
|
||||
}
|
||||
|
||||
@@ -3,5 +3,7 @@
|
||||
pub mod ip;
|
||||
pub mod time;
|
||||
|
||||
#[allow(unused_imports)]
|
||||
pub use ip::*;
|
||||
#[allow(unused_imports)]
|
||||
pub use time::*;
|
||||
@@ -4,11 +4,14 @@ use std::time::Duration;
|
||||
use chrono::{DateTime, Utc};
|
||||
use tracing::{debug, warn, error};
|
||||
|
||||
#[allow(dead_code)]
|
||||
const TIME_SYNC_URL: &str = "https://core.telegram.org/getProxySecret";
|
||||
#[allow(dead_code)]
|
||||
const MAX_TIME_SKEW_SECS: i64 = 30;
|
||||
|
||||
/// Time sync result
|
||||
#[derive(Debug, Clone)]
|
||||
#[allow(dead_code)]
|
||||
pub struct TimeSyncResult {
|
||||
pub server_time: DateTime<Utc>,
|
||||
pub local_time: DateTime<Utc>,
|
||||
@@ -17,6 +20,7 @@ pub struct TimeSyncResult {
|
||||
}
|
||||
|
||||
/// Check time synchronization with Telegram servers
|
||||
#[allow(dead_code)]
|
||||
pub async fn check_time_sync() -> Option<TimeSyncResult> {
|
||||
let client = reqwest::Client::builder()
|
||||
.timeout(Duration::from_secs(10))
|
||||
@@ -60,17 +64,18 @@ pub async fn check_time_sync() -> Option<TimeSyncResult> {
|
||||
}
|
||||
|
||||
/// Background time sync task
|
||||
#[allow(dead_code)]
|
||||
pub async fn time_sync_task(check_interval: Duration) -> ! {
|
||||
loop {
|
||||
if let Some(result) = check_time_sync().await {
|
||||
if result.is_skewed {
|
||||
error!(
|
||||
"System clock is off by {} seconds. Please sync your clock.",
|
||||
result.skew_secs
|
||||
);
|
||||
}
|
||||
if let Some(result) = check_time_sync().await
|
||||
&& result.is_skewed
|
||||
{
|
||||
error!(
|
||||
"System clock is off by {} seconds. Please sync your clock.",
|
||||
result.skew_secs
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
tokio::time::sleep(check_interval).await;
|
||||
}
|
||||
}
|
||||
@@ -172,7 +172,7 @@ zabbix_export:
|
||||
preprocessing:
|
||||
- type: PROMETHEUS_PATTERN
|
||||
parameters:
|
||||
- 'telemt_user_connections_current{user=~"{#TELEMT_USER}"}'
|
||||
- 'telemt_user_connections_current{user="{#TELEMT_USER}"}'
|
||||
- value
|
||||
- ''
|
||||
master_item:
|
||||
@@ -188,7 +188,7 @@ zabbix_export:
|
||||
preprocessing:
|
||||
- type: PROMETHEUS_PATTERN
|
||||
parameters:
|
||||
- 'telemt_user_msgs_from_client{user=~"{#TELEMT_USER}"}'
|
||||
- 'telemt_user_msgs_from_client{user="{#TELEMT_USER}"}'
|
||||
- value
|
||||
- ''
|
||||
master_item:
|
||||
@@ -204,7 +204,7 @@ zabbix_export:
|
||||
preprocessing:
|
||||
- type: PROMETHEUS_PATTERN
|
||||
parameters:
|
||||
- 'telemt_user_msgs_to_client{user=~"{#TELEMT_USER}"}'
|
||||
- 'telemt_user_msgs_to_client{user="{#TELEMT_USER}"}'
|
||||
- value
|
||||
- ''
|
||||
master_item:
|
||||
@@ -221,7 +221,7 @@ zabbix_export:
|
||||
preprocessing:
|
||||
- type: PROMETHEUS_PATTERN
|
||||
parameters:
|
||||
- 'telemt_user_octets_from_client{user=~"{#TELEMT_USER}"}'
|
||||
- 'telemt_user_octets_from_client{user="{#TELEMT_USER}"}'
|
||||
- value
|
||||
- ''
|
||||
master_item:
|
||||
@@ -238,7 +238,7 @@ zabbix_export:
|
||||
preprocessing:
|
||||
- type: PROMETHEUS_PATTERN
|
||||
parameters:
|
||||
- 'telemt_user_octets_to_client{user=~"{#TELEMT_USER}"}'
|
||||
- 'telemt_user_octets_to_client{user="{#TELEMT_USER}"}'
|
||||
- value
|
||||
- ''
|
||||
master_item:
|
||||
@@ -254,7 +254,7 @@ zabbix_export:
|
||||
preprocessing:
|
||||
- type: PROMETHEUS_PATTERN
|
||||
parameters:
|
||||
- 'telemt_user_connections_total{user=~"{#TELEMT_USER}"}'
|
||||
- 'telemt_user_connections_total{user="{#TELEMT_USER}"}'
|
||||
- value
|
||||
- ''
|
||||
master_item:
|
||||
|
||||
Reference in New Issue
Block a user