diff --git a/docs/Config_params/CONFIG_PARAMS.en.md b/docs/Config_params/CONFIG_PARAMS.en.md index ed7c3c2..adf1926 100644 --- a/docs/Config_params/CONFIG_PARAMS.en.md +++ b/docs/Config_params/CONFIG_PARAMS.en.md @@ -219,6 +219,7 @@ This document lists all configuration keys accepted by `config.toml`. | [`ntp_servers`](#cfg-general-ntp_servers) | `String[]` | `["pool.ntp.org"]` | | [`auto_degradation_enabled`](#cfg-general-auto_degradation_enabled) | `bool` | `true` | | [`degradation_min_unavailable_dc_groups`](#cfg-general-degradation_min_unavailable_dc_groups) | `u8` | `2` | +| [`rst_on_close`](#cfg-general-rst_on_close) | `"off"`, `"errors"`, or `"always"` | `"off"` | ## "cfg-general-data_path" - `data_path` @@ -1592,7 +1593,21 @@ This document lists all configuration keys accepted by `config.toml`. [general] degradation_min_unavailable_dc_groups = 2 ``` +## "cfg-general-rst_on_close" +- `rst_on_close` + - **Constraints / validation**: one of `"off"`, `"errors"`, `"always"`. + - **Description**: Controls `SO_LINGER(0)` behaviour on accepted client TCP sockets. + High-traffic proxy servers accumulate `FIN-WAIT-1` and orphaned sockets from connections that never complete the Telegram handshake (scanners, DPI probes, bots). + This option allows sending an immediate `RST` instead of a graceful `FIN` for such connections, freeing kernel resources instantly. + - `"off"` — default. Normal `FIN` on all closes; no behaviour change. + - `"errors"` — `SO_LINGER(0)` is set on `accept()`. If the client successfully completes authentication, linger is cleared and the relay session closes gracefully with `FIN`. Connections closed before handshake completion (timeouts, bad crypto, scanners) send `RST`. + - `"always"` — `SO_LINGER(0)` is set on `accept()` and never cleared. All closes send `RST` regardless of handshake outcome. + - **Example**: + ```toml + [general] + rst_on_close = "errors" + ``` # [general.modes] diff --git a/docs/Config_params/CONFIG_PARAMS.ru.md b/docs/Config_params/CONFIG_PARAMS.ru.md index 4302ba6..fd56556 100644 --- a/docs/Config_params/CONFIG_PARAMS.ru.md +++ b/docs/Config_params/CONFIG_PARAMS.ru.md @@ -219,6 +219,7 @@ | [`ntp_servers`](#cfg-general-ntp_servers) | `String[]` | `["pool.ntp.org"]` | | [`auto_degradation_enabled`](#cfg-general-auto_degradation_enabled) | `bool` | `true` | | [`degradation_min_unavailable_dc_groups`](#cfg-general-degradation_min_unavailable_dc_groups) | `u8` | `2` | +| [`rst_on_close`](#cfg-general-rst_on_close) | `"off"`, `"errors"` или `"always"` | `"off"` | ## "cfg-general-data_path" - `data_path` @@ -1592,7 +1593,21 @@ [general] degradation_min_unavailable_dc_groups = 2 ``` +## "cfg-general-rst_on_close" +- `rst_on_close` + - **Ограничения / валидация**: одно из `"off"`, `"errors"`, `"always"`. + - **Описание**: Управляет поведением `SO_LINGER(0)` на принятых клиентских TCP-сокетах. + На высоконагруженных прокси-серверах накапливаются `FIN-WAIT-1` и осиротевшие (orphan) сокеты от соединений, которые не завершают Telegram-рукопожатие (сканеры, DPI-зонды, боты). + Эта опция позволяет отправлять немедленный `RST` вместо корректного `FIN` для таких соединений, мгновенно освобождая ресурсы ядра. + - `"off"` — по умолчанию. Обычный `FIN` при закрытии всех соединений; поведение не меняется. + - `"errors"` — `SO_LINGER(0)` устанавливается при `accept()`. Если клиент успешно проходит аутентификацию, linger сбрасывается и relay-сессия закрывается корректно через `FIN`. Соединения, закрытые до завершения рукопожатия (таймауты, ошибки крипто, сканеры), отправляют `RST`. + - `"always"` — `SO_LINGER(0)` устанавливается при `accept()` и никогда не сбрасывается. Все закрытия отправляют `RST` независимо от результата рукопожатия. + - **Пример**: + ```toml + [general] + rst_on_close = "errors" + ``` # [general.modes] diff --git a/docs/Setup_examples/VPS_DOUBLE_HOP.en.md b/docs/Setup_examples/VPS_DOUBLE_HOP.en.md index 6b6abe5..093239b 100644 --- a/docs/Setup_examples/VPS_DOUBLE_HOP.en.md +++ b/docs/Setup_examples/VPS_DOUBLE_HOP.en.md @@ -163,7 +163,7 @@ PING 10.10.10.1 (10.10.10.1) 56(84) bytes of data. --- ## Step 2. Installing telemt on Server B (conditionally Netherlands) -Installation and configuration are described [here](https://github.com/telemt/telemt/blob/main/docs/QUICK_START_GUIDE.ru.md) or [here](https://gitlab.com/An0nX/telemt-docker#-quick-start-docker-compose).\ +Installation and configuration are described [here](https://github.com/telemt/telemt/blob/main/docs/Quick_start/QUICK_START_GUIDE.en.md) or [here](https://gitlab.com/An0nX/telemt-docker#-quick-start-docker-compose).\ It is assumed that telemt expects connections on port `443\tcp`. In the telemt config, you must enable the `Proxy` protocol and restrict connections to it only through the tunnel. diff --git a/docs/Setup_examples/VPS_DOUBLE_HOP.ru.md b/docs/Setup_examples/VPS_DOUBLE_HOP.ru.md index 037dfcb..8e22c6c 100644 --- a/docs/Setup_examples/VPS_DOUBLE_HOP.ru.md +++ b/docs/Setup_examples/VPS_DOUBLE_HOP.ru.md @@ -166,7 +166,7 @@ PING 10.10.10.1 (10.10.10.1) 56(84) bytes of data. ## Шаг 2. Установка telemt на Сервере B (_условно Нидерланды_) -Установка и настройка описаны [здесь](https://github.com/telemt/telemt/blob/main/docs/QUICK_START_GUIDE.ru.md) или [здесь](https://gitlab.com/An0nX/telemt-docker#-quick-start-docker-compose).\ +Установка и настройка описаны [здесь](https://github.com/telemt/telemt/blob/main/docs/Quick_start/QUICK_START_GUIDE.ru.md) или [здесь](https://gitlab.com/An0nX/telemt-docker#-quick-start-docker-compose).\ Подразумевается что telemt ожидает подключения на порту `443\tcp`. В конфиге telemt необходимо включить протокол `Proxy` и ограничить подключения к нему только через туннель. diff --git a/docs/Setup_examples/XRAY_DOUBLE_HOP.en.md b/docs/Setup_examples/XRAY_DOUBLE_HOP.en.md new file mode 100644 index 0000000..6cde329 --- /dev/null +++ b/docs/Setup_examples/XRAY_DOUBLE_HOP.en.md @@ -0,0 +1,273 @@ + + +## Concept +- **Server A** (_e.g., RU_):\ + Entry point, accepts Telegram proxy user traffic via **Xray** (port `443\tcp`)\ + and sends it through the tunnel to Server **B**.\ + Public port for Telegram clients — `443\tcp` +- **Server B** (_e.g., NL_):\ + Exit point, runs the **Xray server** (to terminate the tunnel entry point) and **telemt**.\ + The server must have unrestricted access to Telegram Data Centers.\ + Public port for VLESS/REALITY (incoming) — `443\tcp`\ + Internal telemt port (where decrypted Xray traffic ends up) — `8443\tcp` + +The tunnel works over the `VLESS-XTLS-Reality` (or `VLESS/xhttp/reality`) protocol. The original client IP address is preserved thanks to the PROXYv2 protocol, which Xray on Server A dynamically injects via a local loopback before wrapping the traffic into Reality, transparently delivering the real IPs to telemt on Server B. + +--- + +## Step 1. Setup Xray Tunnel (A <-> B) + +You must install **Xray-core** (version 1.8.4 or newer recommended) on both servers. +Official installation script (run on both servers): +```bash +bash -c "$(curl -L https://github.com/XTLS/Xray-install/raw/main/install-release.sh)" @ install +``` + +### Key and Parameter Generation (Run Once) +For configuration, you need a unique UUID and Xray Reality keys. Run on any server with Xray installed: +1. **Client UUID:** +```bash +xray uuid +# Save the output (e.g.: 12345678-abcd-1234-abcd-1234567890ab) — this is +``` +2. **X25519 Keypair (Private & Public) for Reality:** +```bash +xray x25519 +# Save the Private key () and Public key () +``` +3. **Short ID (Reality identifier):** +```bash +openssl rand -hex 16 +# Save the output (e.g.: 0123456789abcdef0123456789abcdef) — this is +``` +4. **Random Path (for xhttp):** +```bash +openssl rand -hex 8 +# Save the output (e.g., abc123def456) to replace in configs +``` + +--- + +### Configuration for Server B (_EU_): + +Create or edit the file `/usr/local/etc/xray/config.json`. +This Xray instance will listen on the public `443` port and proxy valid Reality traffic, while routing "disguised" traffic (e.g., direct web browser scans) to `yahoo.com`. + +```bash +nano /usr/local/etc/xray/config.json +``` + +File content: +```json +{ + "log": { + "loglevel": "error", + "access": "none" + }, + "inbounds": [ + { + "tag": "vless-in", + "port": 443, + "protocol": "vless", + "settings": { + "clients": [ + { + "id": "" + } + ], + "decryption": "none" + }, + "streamSettings": { + "network": "xhttp", + "security": "reality", + "realitySettings": { + "dest": "yahoo.com:443", + "serverNames": [ + "yahoo.com" + ], + "privateKey": "", + "shortIds": [ + "" + ] + }, + "xhttpSettings": { + "path": "/", + "mode": "auto" + } + } + } + ], + "outbounds": [ + { + "tag": "tunnel-to-telemt", + "protocol": "freedom", + "settings": { + "destination": "127.0.0.1:8443" + } + } + ], + "routing": { + "domainStrategy": "AsIs", + "rules": [ + { + "type": "field", + "inboundTag": [ + "vless-in" + ], + "outboundTag": "tunnel-to-telemt" + } + ] + } +} +``` + +Open the firewall port (if enabled): +```bash +sudo ufw allow 443/tcp +``` +Restart and setup Xray to run at boot: +```bash +sudo systemctl restart xray +sudo systemctl enable xray +``` + +--- + +### Configuration for Server A (_RU_): + +Similarly, edit `/usr/local/etc/xray/config.json`. +Here Xray acts as the public entry point: it listens on `443\tcp`, uses a local loopback (via internal port `10444`) to prepend the `PROXYv2` header, and encapsulates the payload via Reality to Server B, instructing Server B to deliver it to its *local* `127.0.0.1:8443` port (where telemt will listen). + +```bash +nano /usr/local/etc/xray/config.json +``` + +File content: +```json +{ + "log": { + "loglevel": "error", + "access": "none" + }, + "inbounds": [ + { + "tag": "public-in", + "port": 443, + "listen": "0.0.0.0", + "protocol": "dokodemo-door", + "settings": { + "address": "127.0.0.1", + "port": 10444, + "network": "tcp" + } + }, + { + "tag": "tunnel-in", + "port": 10444, + "listen": "127.0.0.1", + "protocol": "dokodemo-door", + "settings": { + "address": "127.0.0.1", + "port": 8443, + "network": "tcp" + } + } + ], + "outbounds": [ + { + "tag": "local-injector", + "protocol": "freedom", + "settings": { + "proxyProtocol": 2 + } + }, + { + "tag": "vless-out", + "protocol": "vless", + "settings": { + "vnext": [ + { + "address": "", + "port": 443, + "users": [ + { + "id": "", + "encryption": "none" + } + ] + } + ] + }, + "streamSettings": { + "network": "xhttp", + "security": "reality", + "realitySettings": { + "serverName": "yahoo.com", + "publicKey": "", + "shortId": "", + "spiderX": "/", + "fingerprint": "chrome" + }, + "xhttpSettings": { + "path": "/" + } + } + } + ], + "routing": { + "domainStrategy": "AsIs", + "rules": [ + { + "type": "field", + "inboundTag": ["public-in"], + "outboundTag": "local-injector" + }, + { + "type": "field", + "inboundTag": ["tunnel-in"], + "outboundTag": "vless-out" + } + ] + } +} +``` +*Replace `` with the public IP address of Server B.* + +Open the firewall port for clients (if enabled): +```bash +sudo ufw allow 443/tcp +``` + +Restart and setup Xray to run at boot: +```bash +sudo systemctl restart xray +sudo systemctl enable xray +``` + +--- + +## Step 2. Install telemt on Server B (_EU_) + +telemt installation is heavily covered in the [Quick Start Guide](../Quick_start/QUICK_START_GUIDE.en.md). +By contrast to standard setups, telemt must listen strictly _locally_ (since Xray occupies the public `443` interface) and must expect `PROXYv2` packets. + +Edit the configuration file (`config.toml`) on Server B accordingly: + +```toml +[server] +port = 8443 +listen_addr_ipv4 = "127.0.0.1" +proxy_protocol = true + +[general.links] +show = "*" +public_host = "" +public_port = 443 +``` + +- Address `127.0.0.1` and `port = 8443` instructs the core proxy router to process connections unpacked locally via Xray-server. +- `proxy_protocol = true` commands telemt to parse the injected PROXY header (from Server A's Xray local loopback) and log genuine end-user IPs. +- Under `public_host`, place Server A's public IP address or FQDN to ensure working links are generated for Telegram users. + +Restart `telemt`. Your server is now robust against DPI scanners, passing traffic optimally. + diff --git a/docs/Setup_examples/XRAY_DOUBLE_HOP.ru.md b/docs/Setup_examples/XRAY_DOUBLE_HOP.ru.md new file mode 100644 index 0000000..8cbdf3a --- /dev/null +++ b/docs/Setup_examples/XRAY_DOUBLE_HOP.ru.md @@ -0,0 +1,272 @@ + + +## Концепция +- **Сервер A** (_РФ_):\ + Точка входа, принимает трафик пользователей Telegram-прокси напрямую через **Xray** (порт `443\tcp`)\ + и отправляет его в туннель на Сервер **B**.\ + Порт для клиентов Telegram — `443\tcp` +- **Сервер B** (_условно Нидерланды_):\ + Точка выхода, на нем работает **Xray-сервер** (принимает подключения точки входа) и **telemt**.\ + На сервере должен быть неограниченный доступ до серверов Telegram.\ + Порт для VLESS/REALITY (вход) — `443\tcp`\ + Внутренний порт telemt (куда пробрасывается трафик) — `8443\tcp` + +Туннель работает по протоколу VLESS-XTLS-Reality (или VLESS/xhttp/reality). Оригинальный IP-адрес клиента сохраняется благодаря протоколу PROXYv2, который Xray на Сервере А добавляет через локальный loopback перед упаковкой в туннель, благодаря чему прозрачно доходит до telemt. + +--- + +## Шаг 1. Настройка туннеля Xray (A <-> B) + +На обоих серверах необходимо установить **Xray-core** (рекомендуется версия 1.8.4 или новее). +Официальный скрипт установки (выполнить на обоих серверах): +```bash +bash -c "$(curl -L https://github.com/XTLS/Xray-install/raw/main/install-release.sh)" @ install +``` + +### Генерация ключей и параметров (выполнить один раз) +Для конфигурации потребуются уникальные ID и ключи Xray Reality. Выполните на любом сервере с установленным Xray: +1. **UUID клиента:** +```bash +xray uuid +# Сохраните вывод (например: 12345678-abcd-1234-abcd-1234567890ab) — это +``` +2. **Пара ключей X25519 (Private & Public) для Reality:** +```bash +xray x25519 +# Сохраните Private key () и Public key () +``` +3. **Short ID (идентификатор Reality):** +```bash +openssl rand -hex 16 +# Сохраните вывод (например: 0123456789abcdef0123456789abcdef) — это +``` +4. **Random Path (путь для xhttp):** +```bash +openssl rand -hex 8 +# Сохраните вывод (например, abc123def456), чтобы заменить в конфигах +``` + +--- + +### Конфигурация Сервера B (_Нидерланды_): + +Создаем или редактируем файл `/usr/local/etc/xray/config.json`. +Этот Xray-сервер будет слушать порт `443` и прозрачно пропускать валидный Reality трафик дальше, а "замаскированный" трафик (например, если кто-то стучится в лоб веб-браузером) пойдет на `yahoo.com`. + +```bash +nano /usr/local/etc/xray/config.json +``` + +Содержимое файла: +```json +{ + "log": { + "loglevel": "error", + "access": "none" + }, + "inbounds": [ + { + "tag": "vless-in", + "port": 443, + "protocol": "vless", + "settings": { + "clients": [ + { + "id": "" + } + ], + "decryption": "none" + }, + "streamSettings": { + "network": "xhttp", + "security": "reality", + "realitySettings": { + "dest": "yahoo.com:443", + "serverNames": [ + "yahoo.com" + ], + "privateKey": "", + "shortIds": [ + "" + ] + }, + "xhttpSettings": { + "path": "/", + "mode": "auto" + } + } + } + ], + "outbounds": [ + { + "tag": "tunnel-to-telemt", + "protocol": "freedom", + "settings": { + "destination": "127.0.0.1:8443" + } + } + ], + "routing": { + "domainStrategy": "AsIs", + "rules": [ + { + "type": "field", + "inboundTag": [ + "vless-in" + ], + "outboundTag": "tunnel-to-telemt" + } + ] + } +} +``` + +Открываем порт на фаерволе (если включен): +```bash +sudo ufw allow 443/tcp +``` +Перезапускаем Xray: +```bash +sudo systemctl restart xray +sudo systemctl enable xray +``` + +--- + +### Конфигурация Сервера A (_РФ_): + +Аналогично, редактируем `/usr/local/etc/xray/config.json`. +Здесь Xray выступает публичной точкой: он принимает трафик на внешний порт `443\tcp`, пропускает через локальный loopback (порт `10444`) для добавления PROXYv2-заголовка, и упаковывает в Reality до Сервера B, прося тот доставить данные на *свой локальный* порт `127.0.0.1:8443` (именно там будет слушать telemt). + +```bash +nano /usr/local/etc/xray/config.json +``` + +Содержимое файла: +```json +{ + "log": { + "loglevel": "error", + "access": "none" + }, + "inbounds": [ + { + "tag": "public-in", + "port": 443, + "listen": "0.0.0.0", + "protocol": "dokodemo-door", + "settings": { + "address": "127.0.0.1", + "port": 10444, + "network": "tcp" + } + }, + { + "tag": "tunnel-in", + "port": 10444, + "listen": "127.0.0.1", + "protocol": "dokodemo-door", + "settings": { + "address": "127.0.0.1", + "port": 8443, + "network": "tcp" + } + } + ], + "outbounds": [ + { + "tag": "local-injector", + "protocol": "freedom", + "settings": { + "proxyProtocol": 2 + } + }, + { + "tag": "vless-out", + "protocol": "vless", + "settings": { + "vnext": [ + { + "address": "", + "port": 443, + "users": [ + { + "id": "", + "encryption": "none" + } + ] + } + ] + }, + "streamSettings": { + "network": "xhttp", + "security": "reality", + "realitySettings": { + "serverName": "yahoo.com", + "publicKey": "", + "shortId": "", + "spiderX": "/", + "fingerprint": "chrome" + }, + "xhttpSettings": { + "path": "/" + } + } + } + ], + "routing": { + "domainStrategy": "AsIs", + "rules": [ + { + "type": "field", + "inboundTag": ["public-in"], + "outboundTag": "local-injector" + }, + { + "type": "field", + "inboundTag": ["tunnel-in"], + "outboundTag": "vless-out" + } + ] + } +} +``` +*Замените `` на внешний IP-адрес Сервера B.* + +Открываем порт на фаерволе для клиентов: +```bash +sudo ufw allow 443/tcp +``` + +Перезапускаем Xray: +```bash +sudo systemctl restart xray +sudo systemctl enable xray +``` + +--- + +## Шаг 2. Установка и настройка telemt на Сервере B (_Нидерланды_) + +Установка telemt описана [в основной инструкции](../Quick_start/QUICK_START_GUIDE.ru.md). +Отличие в том, что telemt должен слушать *внутренний* порт (так как 443 занят Xray-сервером), а также ожидать `PROXY` протокол из Xray туннеля. + +В конфиге `config.toml` прокси (на Сервере B) укажите: +```toml +[server] +port = 8443 +listen_addr_ipv4 = "127.0.0.1" +proxy_protocol = true + +[general.links] +show = "*" +public_host = "" +public_port = 443 +``` + +- `port = 8443` и `listen_addr_ipv4 = "127.0.0.1"` означают, что telemt принимает подключения только изнутри (приходящие от локального Xray-процесса). +- `proxy_protocol = true` заставляет telemt парсить PROXYv2-заголовок (который добавил Xray на Сервере A через loopback), восстанавливая IP-адрес конечного пользователя (РФ). +- В `public_host` укажите публичный IP-адрес или домен Сервера A, чтобы ссылки на подключение генерировались корректно. + +Перезапустите `telemt`, и клиенты смогут подключаться по выданным ссылкам. + diff --git a/src/config/types.rs b/src/config/types.rs index 0a5af21..98c22a6 100644 --- a/src/config/types.rs +++ b/src/config/types.rs @@ -159,6 +159,21 @@ impl MeBindStaleMode { } } +/// RST-on-close mode for accepted client sockets. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)] +#[serde(rename_all = "lowercase")] +pub enum RstOnCloseMode { + /// Normal FIN on all closes (default, no behaviour change). + #[default] + Off, + /// SO_LINGER(0) on accept; cleared after successful auth. + /// Pre-handshake failures (scanners, DPI, timeouts) send RST; + /// authenticated relay sessions close gracefully with FIN. + Errors, + /// SO_LINGER(0) on accept, never cleared — all closes send RST. + Always, +} + /// Middle-End writer floor policy mode. #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)] #[serde(rename_all = "lowercase")] @@ -925,6 +940,14 @@ pub struct GeneralConfig { /// Minimum unavailable ME DC groups before degrading. #[serde(default = "default_degradation_min_unavailable_dc_groups")] pub degradation_min_unavailable_dc_groups: u8, + + /// RST-on-close mode for accepted client sockets. + /// `off` — normal FIN on all closes (default). + /// `errors` — SO_LINGER(0) on accept, cleared after successful auth; + /// pre-handshake failures send RST, relayed sessions close gracefully. + /// `always` — SO_LINGER(0) on accept, never cleared; all closes send RST. + #[serde(default)] + pub rst_on_close: RstOnCloseMode, } impl Default for GeneralConfig { @@ -1086,6 +1109,7 @@ impl Default for GeneralConfig { ntp_servers: default_ntp_servers(), auto_degradation_enabled: default_true(), degradation_min_unavailable_dc_groups: default_degradation_min_unavailable_dc_groups(), + rst_on_close: RstOnCloseMode::default(), } } } diff --git a/src/maestro/listeners.rs b/src/maestro/listeners.rs index 96d4cd9..f032d77 100644 --- a/src/maestro/listeners.rs +++ b/src/maestro/listeners.rs @@ -9,7 +9,7 @@ use tokio::net::UnixListener; use tokio::sync::{Semaphore, watch}; use tracing::{debug, error, info, warn}; -use crate::config::ProxyConfig; +use crate::config::{ProxyConfig, RstOnCloseMode}; use crate::crypto::SecureRandom; use crate::ip_tracker::UserIpTracker; use crate::proxy::ClientHandler; @@ -21,6 +21,7 @@ use crate::stats::{ReplayChecker, Stats}; use crate::stream::BufferPool; use crate::tls_front::TlsFrontCache; use crate::transport::middle_proxy::MePool; +use crate::transport::socket::set_linger_zero; use crate::transport::{ListenOptions, UpstreamManager, create_listener, find_listener_processes}; use super::helpers::{is_expected_handshake_eof, print_proxy_links}; @@ -380,6 +381,15 @@ pub(crate) fn spawn_tcp_accept_loops( loop { match listener.accept().await { Ok((stream, peer_addr)) => { + let rst_mode = config_rx.borrow().general.rst_on_close; + #[cfg(unix)] + let raw_fd = { + use std::os::unix::io::AsRawFd; + stream.as_raw_fd() + }; + if matches!(rst_mode, RstOnCloseMode::Errors | RstOnCloseMode::Always) { + let _ = set_linger_zero(&stream); + } if !*admission_rx_tcp.borrow() { debug!(peer = %peer_addr, "Admission gate closed, dropping connection"); drop(stream); @@ -454,6 +464,9 @@ pub(crate) fn spawn_tcp_accept_loops( shared, proxy_protocol_enabled, real_peer_report_for_handler, + #[cfg(unix)] + raw_fd, + rst_mode, ) .run() .await diff --git a/src/proxy/client.rs b/src/proxy/client.rs index fb73db2..0937a8f 100644 --- a/src/proxy/client.rs +++ b/src/proxy/client.rs @@ -804,6 +804,9 @@ pub struct RunningClientHandler { beobachten: Arc, shared: Arc, proxy_protocol_enabled: bool, + #[cfg(unix)] + raw_fd: std::os::unix::io::RawFd, + rst_on_close: crate::config::RstOnCloseMode, } impl ClientHandler { @@ -825,6 +828,11 @@ impl ClientHandler { proxy_protocol_enabled: bool, real_peer_report: Arc>>, ) -> RunningClientHandler { + #[cfg(unix)] + let raw_fd = { + use std::os::unix::io::AsRawFd; + stream.as_raw_fd() + }; Self::new_with_shared( stream, peer, @@ -842,6 +850,9 @@ impl ClientHandler { ProxySharedState::new(), proxy_protocol_enabled, real_peer_report, + #[cfg(unix)] + raw_fd, + crate::config::RstOnCloseMode::Off, ) } @@ -863,6 +874,8 @@ impl ClientHandler { shared: Arc, proxy_protocol_enabled: bool, real_peer_report: Arc>>, + #[cfg(unix)] raw_fd: std::os::unix::io::RawFd, + rst_on_close: crate::config::RstOnCloseMode, ) -> RunningClientHandler { let normalized_peer = normalize_ip(peer); RunningClientHandler { @@ -883,6 +896,9 @@ impl ClientHandler { beobachten, shared, proxy_protocol_enabled, + #[cfg(unix)] + raw_fd, + rst_on_close, } } } @@ -901,6 +917,10 @@ impl RunningClientHandler { debug!(peer = %peer, error = %e, "Failed to configure client socket"); } + #[cfg(unix)] + let raw_fd = self.raw_fd; + let rst_on_close = self.rst_on_close; + let outcome = match self.do_handshake().await? { Some(outcome) => outcome, None => return Ok(()), @@ -908,7 +928,14 @@ impl RunningClientHandler { // Phase 2: relay (WITHOUT handshake timeout — relay has its own activity timeouts) match outcome { - HandshakeOutcome::NeedsRelay(fut) | HandshakeOutcome::NeedsMasking(fut) => fut.await, + HandshakeOutcome::NeedsRelay(fut) => { + #[cfg(unix)] + if matches!(rst_on_close, crate::config::RstOnCloseMode::Errors) { + let _ = crate::transport::socket::clear_linger_fd(raw_fd); + } + fut.await + } + HandshakeOutcome::NeedsMasking(fut) => fut.await, } } diff --git a/src/transport/socket.rs b/src/transport/socket.rs index 32400f4..b751a30 100644 --- a/src/transport/socket.rs +++ b/src/transport/socket.rs @@ -102,14 +102,29 @@ pub fn configure_client_socket( Ok(()) } -/// Set socket to send RST on close (for masking) -#[allow(dead_code)] +/// Set socket to send RST on close instead of FIN, eliminating +/// FIN-WAIT-1 and orphan socket accumulation on high-churn workloads. pub fn set_linger_zero(stream: &TcpStream) -> Result<()> { let socket = socket2::SockRef::from(stream); socket.set_linger(Some(Duration::ZERO))?; Ok(()) } +/// Restore default linger behaviour (graceful FIN) on a socket +/// identified by its raw file descriptor. Safe to call after +/// `TcpStream::into_split()` because the fd remains valid until +/// both halves are dropped. +#[cfg(unix)] +pub fn clear_linger_fd(fd: std::os::unix::io::RawFd) -> Result<()> { + use std::os::unix::io::BorrowedFd; + // SAFETY: the fd is still open — the caller guarantees the + // TcpStream (or its split halves) is alive. + let borrowed = unsafe { BorrowedFd::borrow_raw(fd) }; + let socket = socket2::SockRef::from(&borrowed); + socket.set_linger(None)?; + Ok(()) +} + /// Create a new TCP socket for outgoing connections #[allow(dead_code)] pub fn create_outgoing_socket(addr: SocketAddr) -> Result { diff --git a/tools/grafana-dashboard.json b/tools/grafana-dashboard.json index cd534f6..6c16b8d 100644 --- a/tools/grafana-dashboard.json +++ b/tools/grafana-dashboard.json @@ -1,14 +1,8 @@ { "apiVersion": "dashboard.grafana.app/v1beta1", - "kind": "Dashboard", + "kind": "DashboardWithAccessInfo", "metadata": { - "annotations": { - "grafana.app/folder": "afd9kjusw2jnkb", - "grafana.app/saved-from-ui": "Grafana v12.4.0-21693836646 (f059795f04)" - }, - "labels": {}, - "name": "pi9trh5", - "namespace": "default" + "name": "Telemt MtProto proxy" }, "spec": { "annotations": { @@ -30,7 +24,20 @@ "editable": true, "fiscalYearStartMonth": 0, "graphTooltip": 0, - "links": [], + "links": [ + { + "asDropdown": false, + "icon": "bolt", + "includeVars": false, + "keepTime": false, + "tags": [], + "targetBlank": true, + "title": "Official GitHub repository", + "tooltip": "Official GitHub repository", + "type": "link", + "url": "https://github.com/telemt/telemt" + } + ], "panels": [ { "collapsed": false, @@ -40,9 +47,9 @@ "x": 0, "y": 0 }, - "id": 5, + "id": 200, "panels": [], - "title": "Common", + "title": "General metrics", "type": "row" }, { @@ -50,22 +57,84 @@ "type": "prometheus", "uid": "${datasource}" }, + "description": "Build version reported by telemt.", "fieldConfig": { "defaults": { - "color": { - "mode": "thresholds" - }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { - "color": "red", + "color": "green", "value": 0 }, + { + "color": "yellow", + "value": 1 + }, + { + "color": "red", + "value": 5 + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 4, + "x": 0, + "y": 1 + }, + "id": 301, + "options": { + "colorMode": "none", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "name", + "wideLayout": true + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "max by (version) (telemt_build_info{job=~\"$job\"})", + "instant": true, + "legendFormat": "{{version}}", + "refId": "A" + } + ], + "title": "Build version", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "telemt process uptime in seconds.", + "fieldConfig": { + "defaults": { + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ { "color": "green", - "value": 300 + "value": 0 } ] }, @@ -74,15 +143,15 @@ "overrides": [] }, "gridPos": { - "h": 8, - "w": 6, - "x": 0, + "h": 4, + "w": 4, + "x": 4, "y": 1 }, - "id": 1, + "id": 309, "options": { - "colorMode": "value", - "graphMode": "area", + "colorMode": "none", + "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", @@ -97,22 +166,15 @@ "textMode": "auto", "wideLayout": true }, - "pluginVersion": "12.4.0-21693836646", + "pluginVersion": "12.4.2", "targets": [ { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, "editorMode": "code", - "expr": "max(telemt_uptime_seconds) by (service)", - "format": "time_series", - "legendFormat": "__auto", - "range": true, + "expr": "max(telemt_uptime_seconds{job=~\"$job\"})", "refId": "A" } ], - "title": "uptime", + "title": "Uptime", "type": "stat" }, { @@ -120,22 +182,16 @@ "type": "prometheus", "uid": "${datasource}" }, + "description": "Configured user count derived from per-user unique IP limit series.", "fieldConfig": { "defaults": { - "color": { - "mode": "thresholds" - }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { - "color": "green", + "color": "blue", "value": 0 - }, - { - "color": "red", - "value": 80 } ] }, @@ -144,15 +200,15 @@ "overrides": [] }, "gridPos": { - "h": 8, - "w": 6, - "x": 6, + "h": 4, + "w": 4, + "x": 8, "y": 1 }, - "id": 2, + "id": 317, "options": { "colorMode": "value", - "graphMode": "area", + "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", @@ -167,22 +223,16 @@ "textMode": "auto", "wideLayout": true }, - "pluginVersion": "12.4.0-21693836646", + "pluginVersion": "12.4.2", "targets": [ { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, "editorMode": "code", - "expr": "max(telemt_connections_total) by (service)", - "format": "time_series", - "legendFormat": "__auto", - "range": true, + "expr": "count(telemt_user_unique_ips_limit{job=~\"$job\"})", + "instant": true, "refId": "A" } ], - "title": "connections_total", + "title": "Configured users", "type": "stat" }, { @@ -190,36 +240,385 @@ "type": "prometheus", "uid": "${datasource}" }, + "description": "Total inbound and outbound user traffic bytes since process start.", "fieldConfig": { "defaults": { - "color": { - "mode": "thresholds" - }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { - "color": "green", + "color": "blue", "value": 0 - }, - { - "color": "red", - "value": 80 } ] }, - "unit": "none" + "unit": "decbytes" }, - "overrides": [] + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "In total" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "blue", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Out total" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "green", + "mode": "fixed" + } + } + ] + } + ] }, "gridPos": { - "h": 8, - "w": 6, + "h": 4, + "w": 4, "x": 12, "y": 1 }, - "id": 3, + "id": 318, + "options": { + "colorMode": "value", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "horizontal", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "value_and_name", + "wideLayout": true + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "sum(telemt_user_octets_from_client{job=~\"$job\"})", + "instant": true, + "legendFormat": "In total", + "refId": "A" + }, + { + "editorMode": "code", + "expr": "sum(telemt_user_octets_to_client{job=~\"$job\"})", + "instant": true, + "legendFormat": "Out total", + "refId": "B" + } + ], + "title": "Total traffic", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Current number of in-use shared buffers.", + "fieldConfig": { + "defaults": { + "color": { + "fixedColor": "blue", + "mode": "fixed" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "blue", + "value": 0 + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 4, + "x": 16, + "y": 1 + }, + "id": 312, + "options": { + "colorMode": "none", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "max(telemt_buffer_pool_buffers_total{kind=\"in_use\",job=~\"$job\"})", + "legendFormat": "in use", + "refId": "A" + } + ], + "title": "Buffer pool state", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "User-series export state (1 enabled, 0 suppressed).", + "fieldConfig": { + "defaults": { + "mappings": [], + "max": 1, + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": 0 + }, + { + "color": "green", + "value": 1 + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 4, + "x": 20, + "y": 1 + }, + "id": 311, + "options": { + "colorMode": "value", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "max(1 - telemt_telemetry_user_series_suppressed{job=~\"$job\"})", + "instant": true, + "refId": "A" + } + ], + "title": "User series enabled", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Accepted client connections in the selected time range.", + "fieldConfig": { + "defaults": { + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": 0 + }, + { + "color": "green", + "value": 1 + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 4, + "x": 0, + "y": 5 + }, + "id": 7, + "options": { + "colorMode": "none", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "sum(increase(telemt_connections_total{job=~\"$job\"}[$__range]))", + "refId": "A" + } + ], + "title": "Accepted connections", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Failed or rejected client connections in the selected time range.", + "fieldConfig": { + "defaults": { + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 1 + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 4, + "x": 4, + "y": 5 + }, + "id": 8, + "options": { + "colorMode": "none", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "sum(increase(telemt_connections_bad_total{job=~\"$job\"}[$__range]))", + "refId": "A" + } + ], + "title": "Bad connections", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Bad connection ratio over the selected interval.", + "fieldConfig": { + "defaults": { + "mappings": [], + "max": 100, + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "yellow", + "value": 1 + }, + { + "color": "red", + "value": 5 + } + ] + }, + "unit": "percent" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 4, + "x": 8, + "y": 5 + }, + "id": 10, "options": { "colorMode": "value", "graphMode": "area", @@ -237,22 +636,15 @@ "textMode": "auto", "wideLayout": true }, - "pluginVersion": "12.4.0-21693836646", + "pluginVersion": "12.4.2", "targets": [ { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, "editorMode": "code", - "expr": "max(telemt_connections_bad_total) by (service)", - "format": "time_series", - "legendFormat": "__auto", - "range": true, + "expr": "100 * sum(rate(telemt_connections_bad_total{job=~\"$job\"}[$__rate_interval])) / clamp_min(sum(rate(telemt_connections_total{job=~\"$job\"}[$__rate_interval])), 1e-9)", "refId": "A" } ], - "title": "connections_bad", + "title": "Bad connection ratio", "type": "stat" }, { @@ -260,10 +652,45 @@ "type": "prometheus", "uid": "${datasource}" }, + "description": "Per-second rates for total, bad, handshake-timeout, and permit-timeout connection events.", "fieldConfig": { "defaults": { "color": { - "mode": "thresholds" + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } }, "mappings": [], "thresholds": { @@ -279,19 +706,160 @@ } ] }, + "unit": "cps" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Connections/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "blue", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Bad/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Timeouts/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Permit waits/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "yellow", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 5 + }, + "id": 11, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "sum(rate(telemt_connections_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "Connections/s", + "refId": "A" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_connections_bad_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "Bad/s", + "refId": "B" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_handshake_timeouts_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "Timeouts/s", + "refId": "C" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_accept_permit_timeout_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "Permit waits/s", + "refId": "D" + } + ], + "title": "Connection and permit rates", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Accepted connections dropped by permit-acquisition timeout in the selected time range.", + "fieldConfig": { + "defaults": { + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "yellow", + "value": 1 + }, + { + "color": "red", + "value": 5 + } + ] + }, "unit": "none" }, "overrides": [] }, "gridPos": { - "h": 8, - "w": 6, - "x": 18, - "y": 1 + "h": 4, + "w": 4, + "x": 0, + "y": 9 }, - "id": 4, + "id": 302, "options": { - "colorMode": "value", + "colorMode": "none", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", @@ -307,22 +875,141 @@ "textMode": "auto", "wideLayout": true }, - "pluginVersion": "12.4.0-21693836646", + "pluginVersion": "12.4.2", "targets": [ { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, "editorMode": "code", - "expr": "max(telemt_handshake_timeouts_total) by (service)", - "format": "time_series", - "legendFormat": "__auto", - "range": true, + "expr": "sum(increase(telemt_accept_permit_timeout_total{job=~\"$job\"}[$__range]))", "refId": "A" } ], - "title": "handshake_timeouts", + "title": "Permit wait timeouts", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Handshake validations that exhausted the authentication candidate budget in the selected time range.", + "fieldConfig": { + "defaults": { + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "yellow", + "value": 1 + }, + { + "color": "red", + "value": 5 + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 4, + "x": 4, + "y": 9 + }, + "id": 303, + "options": { + "colorMode": "none", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "sum(increase(telemt_auth_budget_exhausted_total{job=~\"$job\"}[$__range]))", + "refId": "A" + } + ], + "title": "Auth budget exhausted", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Handshake timeouts in the selected time range.", + "fieldConfig": { + "defaults": { + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 1 + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 4, + "x": 8, + "y": 9 + }, + "id": 9, + "options": { + "colorMode": "none", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "sum(increase(telemt_handshake_timeouts_total{job=~\"$job\"}[$__range]))", + "refId": "A" + } + ], + "title": "Handshake timeouts", "type": "stat" }, { @@ -331,12 +1018,11 @@ "h": 1, "w": 24, "x": 0, - "y": 9 + "y": 13 }, - "id": 6, + "id": 320, "panels": [], - "repeat": "user", - "title": "$user", + "title": "Upstream connectivity", "type": "row" }, { @@ -344,6 +1030,1788 @@ "type": "prometheus", "uid": "${datasource}" }, + "description": "Upstream connect attempt/success/fail rates for Telegram/DC connectivity path.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "bars", + "fillOpacity": 70, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "normal" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cps" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Attempt/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "blue", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Success/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "green", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Fail/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 14 + }, + "id": 12, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "sum(rate(telemt_upstream_connect_attempt_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "Attempt/s", + "refId": "A" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_upstream_connect_success_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "Success/s", + "refId": "B" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_upstream_connect_fail_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "Fail/s", + "refId": "C" + } + ], + "title": "Upstream connect outcomes", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Per-second rates of upstream connect-attempt buckets per request.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cps" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "1 attempt/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "green", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "2 attempts/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "yellow", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "3-4 attempts/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "yellow", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": ">4 attempts/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 14 + }, + "id": 321, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "sum(rate(telemt_upstream_connect_attempts_per_request{bucket=\"1\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "1 attempt/s", + "refId": "A" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_upstream_connect_attempts_per_request{bucket=\"2\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "2 attempts/s", + "refId": "B" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_upstream_connect_attempts_per_request{bucket=\"3_4\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "3-4 attempts/s", + "refId": "C" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_upstream_connect_attempts_per_request{bucket=\"gt_4\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": ">4 attempts/s", + "refId": "D" + } + ], + "title": "Upstream connect attempts per request", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Per-second rates of slow upstream connect buckets and hard-error failfast events.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cps" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "success >1s/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "yellow", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "fail >1s/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "failfast hard/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 14 + }, + "id": 322, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "sum(rate(telemt_upstream_connect_duration_success_total{bucket=\"gt_1000ms\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "success >1s/s", + "refId": "A" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_upstream_connect_duration_fail_total{bucket=\"gt_1000ms\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "fail >1s/s", + "refId": "B" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_upstream_connect_failfast_hard_error_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "failfast hard/s", + "refId": "C" + } + ], + "title": "Upstream connect duration and failfast", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 22 + }, + "id": 330, + "panels": [], + "title": "Authentication and security", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Per-second rates of authentication expensive checks and budget exhaustion.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cps" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Expensive checks/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "blue", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Budget exhausted/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 23 + }, + "id": 304, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "sum(rate(telemt_auth_expensive_checks_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "Expensive checks/s", + "refId": "A" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_auth_budget_exhausted_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "Budget exhausted/s", + "refId": "B" + } + ], + "title": "Auth validation pressure", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Per-second rates of invalid secure padding and permit wait timeouts.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cps" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "secure padding invalid/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "permit wait timeout/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "yellow", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 23 + }, + "id": 331, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "sum(rate(telemt_secure_padding_invalid_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "secure padding invalid/s", + "refId": "A" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_accept_permit_timeout_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "permit wait timeout/s", + "refId": "B" + } + ], + "title": "Security rejects", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 31 + }, + "id": 340, + "panels": [], + "title": "Conntrack control", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Current conntrack-control state flags (enabled, available, pressure_active, rule_apply_ok).", + "fieldConfig": { + "defaults": { + "mappings": [], + "max": 1, + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": 0 + }, + { + "color": "green", + "value": 1 + } + ] + }, + "unit": "none" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "pressure_active" + }, + "properties": [ + { + "id": "thresholds", + "value": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 1 + } + ] + } + } + ] + } + ] + }, + "gridPos": { + "h": 4, + "w": 8, + "x": 0, + "y": 32 + }, + "id": 305, + "options": { + "colorMode": "value", + "graphMode": "none", + "justifyMode": "center", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": false + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "max(telemt_conntrack_control_state{flag=\"enabled\",job=~\"$job\"})", + "instant": true, + "legendFormat": "enabled", + "refId": "A" + }, + { + "editorMode": "code", + "expr": "max(telemt_conntrack_control_state{flag=\"available\",job=~\"$job\"})", + "instant": true, + "legendFormat": "available", + "refId": "B" + }, + { + "editorMode": "code", + "expr": "max(telemt_conntrack_control_state{flag=\"pressure_active\",job=~\"$job\"})", + "instant": true, + "legendFormat": "pressure_active", + "refId": "C" + }, + { + "editorMode": "code", + "expr": "max(telemt_conntrack_control_state{flag=\"rule_apply_ok\",job=~\"$job\"})", + "instant": true, + "legendFormat": "rule_apply_ok", + "refId": "D" + } + ], + "title": "Conntrack state flags", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Per-second rates of conntrack delete outcomes and dropped close events.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cps" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Delete attempt/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "blue", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Delete success/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "green", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Delete not_found/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6B7280", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Delete error/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Close-event drop/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 16, + "x": 8, + "y": 32 + }, + "id": 306, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "sum(rate(telemt_conntrack_delete_total{result=\"attempt\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "Delete attempt/s", + "refId": "A" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_conntrack_delete_total{result=\"success\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "Delete success/s", + "refId": "B" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_conntrack_delete_total{result=\"not_found\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "Delete not_found/s", + "refId": "C" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_conntrack_delete_total{result=\"error\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "Delete error/s", + "refId": "D" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_conntrack_close_event_drop_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "Close-event drop/s", + "refId": "E" + } + ], + "title": "Conntrack delete and drop rates", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Current conntrack close-event queue depth.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "yellow", + "value": 1 + }, + { + "color": "red", + "value": 10 + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 8, + "x": 0, + "y": 36 + }, + "id": 307, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "max(telemt_conntrack_event_queue_depth{job=~\"$job\"})", + "legendFormat": "Queue depth", + "refId": "A" + } + ], + "title": "Conntrack event queue depth", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 40 + }, + "id": 310, + "panels": [], + "title": "ME metrics", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Current ME telemetry mode level.", + "fieldConfig": { + "defaults": { + "mappings": [ + { + "options": { + "0": { + "color": "#6B7280", + "text": "silent" + }, + "1": { + "color": "green", + "text": "normal" + }, + "2": { + "color": "orange", + "text": "debug" + } + }, + "type": "value" + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "#6B7280", + "value": 0 + }, + { + "color": "green", + "value": 1 + }, + { + "color": "orange", + "value": 2 + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 4, + "x": 0, + "y": 41 + }, + "id": 316, + "options": { + "colorMode": "value", + "graphMode": "none", + "justifyMode": "center", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "value", + "wideLayout": false + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "max(telemt_telemetry_me_level{level=\"normal\",job=~\"$job\"}) + 2 * max(telemt_telemetry_me_level{level=\"debug\",job=~\"$job\"})", + "instant": true, + "refId": "A" + } + ], + "title": "ME telemetry mode", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "ME handshake rejects in the selected time range.", + "fieldConfig": { + "defaults": { + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "yellow", + "value": 1 + }, + { + "color": "red", + "value": 5 + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 4, + "x": 4, + "y": 41 + }, + "id": 308, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "sum(increase(telemt_me_handshake_reject_total{job=~\"$job\"}[$__range]))", + "refId": "A" + } + ], + "title": "ME handshake rejects", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Reconnect attempts/success and reader EOF event rates in Middle-End (ME) subsystem.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cps" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Reconnect attempt/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "yellow", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Reconnect success/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "green", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Reader EOF/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 41 + }, + "id": 14, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_reconnect_attempts_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "Reconnect attempt/s", + "refId": "A" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_reconnect_success_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "Reconnect success/s", + "refId": "B" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_reader_eof_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "Reader EOF/s", + "refId": "C" + } + ], + "title": "ME reconnect and reader EOF rates", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Rates of Middle-End (ME) route drops by reason and total crypto desync detections.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cps" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Drop no_conn/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "yellow", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Drop channel_closed/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "yellow", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Drop queue_full/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Desync/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 41 + }, + "id": 15, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_route_drop_no_conn_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "Drop no_conn/s", + "refId": "A" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_route_drop_channel_closed_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "Drop channel_closed/s", + "refId": "B" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_route_drop_queue_full_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "Drop queue_full/s", + "refId": "C" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_desync_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "Desync/s", + "refId": "D" + } + ], + "title": "ME route drops and crypto desync", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Per-second rate of ME handshake rejects, including error-code breakdown.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cps" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "reject total/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 4, + "w": 8, + "x": 0, + "y": 45 + }, + "id": 353, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_handshake_reject_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "reject total/s", + "refId": "A" + }, + { + "editorMode": "code", + "expr": "sum by (error_code) (rate(telemt_me_handshake_error_code_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "Code {{error_code}}/s", + "refId": "B" + } + ], + "title": "ME handshake reject rate by code", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Current counts of active and warm ME writers.", "fieldConfig": { "defaults": { "color": { @@ -399,21 +2867,52 @@ }, "unit": "none" }, - "overrides": [] + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Active" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "green", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Warm" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6B7280", + "mode": "fixed" + } + } + ] + } + ] }, "gridPos": { "h": 8, - "w": 12, + "w": 8, "x": 0, - "y": 10 + "y": 49 }, - "id": 7, + "id": 13, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", - "showLegend": false + "showLegend": true }, "tooltip": { "hideZeros": false, @@ -421,18 +2920,22 @@ "sort": "none" } }, - "pluginVersion": "12.4.0-21693836646", + "pluginVersion": "12.4.2", "targets": [ { "editorMode": "code", - "expr": "sum(telemt_user_connections_total{user=\"$user\"}) by (user)", - "format": "time_series", - "legendFormat": "{{ user }}", - "range": true, + "expr": "sum(telemt_me_writers_active_current{job=~\"$job\"})", + "legendFormat": "Active", "refId": "A" + }, + { + "editorMode": "code", + "expr": "sum(telemt_me_writers_warm_current{job=~\"$job\"})", + "legendFormat": "Warm", + "refId": "B" } ], - "title": "user_connections", + "title": "ME writers state", "type": "timeseries" }, { @@ -440,6 +2943,334 @@ "type": "prometheus", "uid": "${datasource}" }, + "description": "Keepalive sent, pong, failed and timeout rates for ME.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cps" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "sent/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "blue", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "pong/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "green", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "failed/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "timeout/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 16, + "x": 8, + "y": 49 + }, + "id": 313, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_keepalive_sent_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "sent/s", + "refId": "A" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_keepalive_pong_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "pong/s", + "refId": "B" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_keepalive_failed_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "failed/s", + "refId": "C" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_keepalive_timeout_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "timeout/s", + "refId": "D" + } + ], + "title": "ME keepalive health", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Per-second rates of ME pool forced closes and refill events.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cps" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "forced close/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "yellow", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "refill triggered/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "blue", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "refill failed/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 57 + }, + "id": 314, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "sum(rate(telemt_pool_force_close_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "forced close/s", + "refId": "A" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_refill_triggered_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "refill triggered/s", + "refId": "B" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_refill_failed_total{job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "refill failed/s", + "refId": "C" + } + ], + "title": "ME refill activity", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Current ME capacity target, active writers, and warm writers.", "fieldConfig": { "defaults": { "color": { @@ -495,21 +3326,67 @@ }, "unit": "none" }, - "overrides": [] + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "target writers" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "blue", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "active writers" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "green", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "warm writers" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6B7280", + "mode": "fixed" + } + } + ] + } + ] }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 10 + "y": 57 }, - "id": 8, + "id": 315, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", - "showLegend": false + "showLegend": true }, "tooltip": { "hideZeros": false, @@ -517,25 +3394,49 @@ "sort": "none" } }, - "pluginVersion": "12.4.0-21693836646", + "pluginVersion": "12.4.2", "targets": [ { "editorMode": "code", - "expr": "sum(telemt_user_connections_current{user=\"$user\"}) by (user)", - "format": "time_series", - "legendFormat": "{{ user }}", - "range": true, + "expr": "max(telemt_me_adaptive_floor_target_writers_total{job=~\"$job\"})", + "legendFormat": "target writers", "refId": "A" + }, + { + "editorMode": "code", + "expr": "sum(telemt_me_writers_active_current{job=~\"$job\"})", + "legendFormat": "active writers", + "refId": "B" + }, + { + "editorMode": "code", + "expr": "sum(telemt_me_writers_warm_current{job=~\"$job\"})", + "legendFormat": "warm writers", + "refId": "C" } ], - "title": "user_connections_current", + "title": "ME capacity", "type": "timeseries" }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 65 + }, + "id": 355, + "panels": [], + "title": "ME debug metrics", + "type": "row" + }, { "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "description": "Per-second rates of ME debug D2C batch-byte buckets.", "fieldConfig": { "defaults": { "color": { @@ -589,7 +3490,1021 @@ } ] }, - "unit": "binBps" + "unit": "cps" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "0-1 KiB/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "green", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "1-4 KiB/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "blue", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "4-16 KiB/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "yellow", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "16-64 KiB/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "orange", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "64-128 KiB/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F97316", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": ">128 KiB/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 66 + }, + "id": 356, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_d2c_batch_bytes_bucket_total{bucket=\"0_1k\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "0-1 KiB/s", + "refId": "A" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_d2c_batch_bytes_bucket_total{bucket=\"1k_4k\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "1-4 KiB/s", + "refId": "B" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_d2c_batch_bytes_bucket_total{bucket=\"4k_16k\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "4-16 KiB/s", + "refId": "C" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_d2c_batch_bytes_bucket_total{bucket=\"16k_64k\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "16-64 KiB/s", + "refId": "D" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_d2c_batch_bytes_bucket_total{bucket=\"64k_128k\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "64-128 KiB/s", + "refId": "E" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_d2c_batch_bytes_bucket_total{bucket=\"gt_128k\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": ">128 KiB/s", + "refId": "F" + } + ], + "title": "ME debug D2C batch bytes", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Per-second rates of ME debug D2C batch-size buckets.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cps" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "1 frame/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "green", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "2-4 frames/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "blue", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "5-8 frames/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "yellow", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "9-16 frames/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "orange", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "17-32 frames/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F97316", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": ">32 frames/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 66 + }, + "id": 357, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_d2c_batch_frames_bucket_total{bucket=\"1\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "1 frame/s", + "refId": "A" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_d2c_batch_frames_bucket_total{bucket=\"2_4\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "2-4 frames/s", + "refId": "B" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_d2c_batch_frames_bucket_total{bucket=\"5_8\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "5-8 frames/s", + "refId": "C" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_d2c_batch_frames_bucket_total{bucket=\"9_16\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "9-16 frames/s", + "refId": "D" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_d2c_batch_frames_bucket_total{bucket=\"17_32\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "17-32 frames/s", + "refId": "E" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_d2c_batch_frames_bucket_total{bucket=\"gt_32\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": ">32 frames/s", + "refId": "F" + } + ], + "title": "ME debug D2C batch size", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Per-second rates of ME debug D2C flush-duration buckets.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cps" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "0-50 us/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "green", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "51-200 us/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "blue", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "201-1000 us/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "yellow", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "1-5 ms/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "orange", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "5-20 ms/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F97316", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": ">20 ms/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 66 + }, + "id": 358, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_d2c_flush_duration_us_bucket_total{bucket=\"0_50\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "0-50 us/s", + "refId": "A" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_d2c_flush_duration_us_bucket_total{bucket=\"51_200\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "51-200 us/s", + "refId": "B" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_d2c_flush_duration_us_bucket_total{bucket=\"201_1000\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "201-1000 us/s", + "refId": "C" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_d2c_flush_duration_us_bucket_total{bucket=\"1001_5000\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "1-5 ms/s", + "refId": "D" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_d2c_flush_duration_us_bucket_total{bucket=\"5001_20000\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "5-20 ms/s", + "refId": "E" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_me_d2c_flush_duration_us_bucket_total{bucket=\"gt_20000\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": ">20 ms/s", + "refId": "F" + } + ], + "title": "ME debug D2C flush duration", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 74 + }, + "id": 350, + "panels": [], + "title": "Access and limits", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Current active and recent user/IP tracker sizes and cleanup queue depth.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "none" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "users active" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "blue", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "users recent" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6B7280", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "IPs active" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "blue", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "IPs recent" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6B7280", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "cleanup queue" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "yellow", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 75 + }, + "id": 351, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "max(telemt_ip_tracker_users{scope=\"active\",job=~\"$job\"})", + "legendFormat": "users active", + "refId": "A" + }, + { + "editorMode": "code", + "expr": "max(telemt_ip_tracker_users{scope=\"recent\",job=~\"$job\"})", + "legendFormat": "users recent", + "refId": "B" + }, + { + "editorMode": "code", + "expr": "max(telemt_ip_tracker_entries{scope=\"active\",job=~\"$job\"})", + "legendFormat": "IPs active", + "refId": "C" + }, + { + "editorMode": "code", + "expr": "max(telemt_ip_tracker_entries{scope=\"recent\",job=~\"$job\"})", + "legendFormat": "IPs recent", + "refId": "D" + }, + { + "editorMode": "code", + "expr": "max(telemt_ip_tracker_cleanup_queue_len{job=~\"$job\"})", + "legendFormat": "cleanup queue", + "refId": "E" + } + ], + "title": "IP tracker state", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "IP reservation rollbacks in the selected time range.", + "fieldConfig": { + "defaults": { + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 12, + "x": 12, + "y": 75 + }, + "id": 352, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "sum(increase(telemt_ip_reservation_rollback_total{job=~\"$job\"}[$__range]))", + "refId": "A" + } + ], + "title": "IP reservation rollbacks", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Reservation rollback rate by reason.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "cps" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "tcp_limit/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "yellow", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "quota_limit/s" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 4, + "w": 12, + "x": 12, + "y": 79 + }, + "id": 354, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "sum(rate(telemt_ip_reservation_rollback_total{reason=\"tcp_limit\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "tcp_limit/s", + "refId": "A" + }, + { + "editorMode": "code", + "expr": "sum(rate(telemt_ip_reservation_rollback_total{reason=\"quota_limit\",job=~\"$job\"}[$__rate_interval]))", + "legendFormat": "quota_limit/s", + "refId": "B" + } + ], + "title": "IP reservation rollback rate by reason", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Current per-user unique IP utilization percentage.", + "fieldConfig": { + "defaults": { + "mappings": [], + "max": 100, + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "yellow", + "value": 60 + }, + { + "color": "red", + "value": 85 + } + ] + }, + "unit": "percent" }, "overrides": [] }, @@ -597,91 +4512,63 @@ "h": 8, "w": 12, "x": 0, - "y": 18 + "y": 83 }, - "id": 9, + "id": 23, "options": { + "displayMode": "basic", "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": false }, - "tooltip": { - "hideZeros": false, - "mode": "single", - "sort": "none" - } + "maxVizHeight": 300, + "minVizHeight": 16, + "minVizWidth": 8, + "namePlacement": "auto", + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showUnfilled": true, + "sizing": "auto", + "valueMode": "color" }, - "pluginVersion": "12.4.0-21693836646", + "pluginVersion": "12.4.2", "targets": [ { "editorMode": "code", - "expr": "- sum(rate(telemt_user_octets_from_client{user=\"$user\"}[$__rate_interval])) by (user)", - "format": "time_series", - "legendFormat": "{{ user }} TX", - "range": true, + "expr": "sort_desc(100 * max by (user) (telemt_user_unique_ips_utilization{job=~\"$job\",user=~\"$user\"}))", + "instant": true, + "legendFormat": "{{user}}", "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "editorMode": "code", - "expr": "sum(rate(telemt_user_octets_to_client{user=\"$user\"}[$__rate_interval])) by (user)", - "format": "time_series", - "legendFormat": "{{ user }} RX", - "range": true, - "refId": "B" } ], - "title": "user_octets", - "type": "timeseries" + "title": "User unique IP utilization", + "type": "bargauge" }, { "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "description": "Current unique IP count per selected user.", "fieldConfig": { "defaults": { - "color": { - "mode": "palette-classic" - }, "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 0, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false + "align": "auto", + "cellOptions": { + "type": "auto" }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" + "footer": { + "reducers": [] }, - "showPoints": "auto", - "showValues": false, - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } + "inspect": false }, "mappings": [], "thresholds": { @@ -697,7 +4584,7 @@ } ] }, - "unit": "pps" + "unit": "none" }, "overrides": [] }, @@ -705,95 +4592,368 @@ "h": 8, "w": 12, "x": 12, - "y": 18 + "y": 83 }, - "id": 10, + "id": 21, "options": { - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": false - }, - "tooltip": { - "hideZeros": false, - "mode": "single", - "sort": "none" - } + "cellHeight": "sm", + "showHeader": true }, - "pluginVersion": "12.4.0-21693836646", + "pluginVersion": "12.4.2", "targets": [ { "editorMode": "code", - "expr": "- sum(rate(telemt_user_msgs_from_client{user=\"$user\"}[$__rate_interval])) by (user)", - "format": "time_series", - "legendFormat": "{{ user }} TX", - "range": true, + "expr": "sum by (user) (telemt_user_unique_ips_current{job=~\"$job\",user=~\"$user\"})", + "format": "table", + "instant": true, + "refId": "A" + } + ], + "title": "User unique IPs", + "transformations": [ + { + "id": "organize", + "options": { + "excludeByName": { + "Time": true + }, + "renameByName": { + "Value": "Unique IPs", + "user": "User" + } + } + }, + { + "id": "sortBy", + "options": { + "fields": { + "Unique IPs": true + }, + "sort": [ + { + "desc": true, + "field": "Unique IPs" + } + ] + } + } + ], + "type": "table" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 91 + }, + "id": 300, + "panels": [], + "title": "Users", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Per-user active connections, traffic totals, message rates, and traffic rates.", + "fieldConfig": { + "defaults": { + "custom": { + "align": "auto", + "cellOptions": { + "type": "auto" + }, + "footer": { + "reducers": [] + }, + "inspect": false + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "none" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "In Total" + }, + "properties": [ + { + "id": "unit", + "value": "decbytes" + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Out Total" + }, + "properties": [ + { + "id": "unit", + "value": "decbytes" + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "In msg/s" + }, + "properties": [ + { + "id": "unit", + "value": "pps" + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Out msg/s" + }, + "properties": [ + { + "id": "unit", + "value": "pps" + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "In traffic" + }, + "properties": [ + { + "id": "unit", + "value": "binBps" + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Out traffic" + }, + "properties": [ + { + "id": "unit", + "value": "binBps" + } + ] + } + ] + }, + "gridPos": { + "h": 16, + "w": 24, + "x": 0, + "y": 92 + }, + "id": 16, + "options": { + "cellHeight": "sm", + "showHeader": true, + "sortBy": [ + { + "desc": true, + "displayName": "Active Connections" + } + ] + }, + "pluginVersion": "12.4.2", + "targets": [ + { + "editorMode": "code", + "expr": "sum by (user) (telemt_user_connections_current{job=~\"$job\",user=~\"$user\"})", + "format": "table", + "instant": true, "refId": "A" }, { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, "editorMode": "code", - "expr": "sum(rate(telemt_user_msgs_to_client{user=\"$user\"}[$__rate_interval])) by (user)", - "format": "time_series", - "legendFormat": "{{ user }} RX", - "range": true, + "expr": "sum by (user) (telemt_user_octets_from_client{job=~\"$job\",user=~\"$user\"})", + "format": "table", + "instant": true, "refId": "B" + }, + { + "editorMode": "code", + "expr": "sum by (user) (telemt_user_octets_to_client{job=~\"$job\",user=~\"$user\"})", + "format": "table", + "instant": true, + "refId": "C" + }, + { + "editorMode": "code", + "expr": "sum by (user) (rate(telemt_user_msgs_from_client{job=~\"$job\",user=~\"$user\"}[$__rate_interval]))", + "format": "table", + "instant": true, + "refId": "D" + }, + { + "editorMode": "code", + "expr": "sum by (user) (rate(telemt_user_msgs_to_client{job=~\"$job\",user=~\"$user\"}[$__rate_interval]))", + "format": "table", + "instant": true, + "refId": "E" + }, + { + "editorMode": "code", + "expr": "sum by (user) (rate(telemt_user_octets_from_client{job=~\"$job\",user=~\"$user\"}[$__rate_interval]))", + "format": "table", + "instant": true, + "refId": "F" + }, + { + "editorMode": "code", + "expr": "sum by (user) (rate(telemt_user_octets_to_client{job=~\"$job\",user=~\"$user\"}[$__rate_interval]))", + "format": "table", + "instant": true, + "refId": "G" } ], - "title": "user_msgs", - "type": "timeseries" + "title": "User connections, traffic, and message rates", + "transformations": [ + { + "id": "joinByField", + "options": { + "byField": "user", + "mode": "outer" + } + }, + { + "id": "organize", + "options": { + "excludeByName": { + "Time": true, + "Time #A": true, + "Time #B": true, + "Time #C": true, + "Time #D": true, + "Time #E": true, + "Time #F": true, + "Time #G": true, + "Time 1": true, + "Time 2": true, + "Time 3": true, + "Time 4": true, + "Time 5": true, + "Time 6": true + }, + "renameByName": { + "Value": "Active Connections", + "Value #A": "Active Connections", + "Value #B": "In Total", + "Value #C": "Out Total", + "Value #D": "In msg/s", + "Value #E": "Out msg/s", + "Value #F": "In traffic", + "Value #G": "Out traffic", + "Value 1": "In Total", + "Value 2": "Out Total", + "Value 3": "In msg/s", + "Value 4": "Out msg/s", + "Value 5": "In traffic", + "Value 6": "Out traffic", + "user": "User" + } + } + }, + { + "id": "sortBy", + "options": { + "fields": { + "Active Connections": true + }, + "sort": [ + { + "desc": true, + "field": "Active Connections" + } + ] + } + } + ], + "type": "table" } ], "preload": false, + "refresh": "30s", "schemaVersion": 42, - "tags": [], + "tags": [ + "telemt", + "mtproto", + "telegram" + ], "templating": { "list": [ { - "current": { - "text": "docker", - "value": "docker" - }, - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "definition": "label_values(telemt_user_connections_total,user)", - "hide": 2, - "multi": true, - "name": "user", - "options": [], - "query": { - "qryType": 1, - "query": "label_values(telemt_user_connections_total,user)", - "refId": "VariableQueryEditor-VariableQuery" - }, - "refresh": 1, - "regex": "", - "regexApplyTo": "value", - "sort": 1, - "type": "query" - }, - { - "current": { - "text": "VM long-term", - "value": "P7D3016A027385E71" - }, + "label": "Datasource", "name": "datasource", "options": [], "query": "prometheus", "refresh": 1, - "regex": "", "type": "datasource" + }, + { + "allValue": ".*", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "definition": "label_values(telemt_build_info, job)", + "includeAll": true, + "label": "Job", + "multi": true, + "name": "job", + "options": [], + "query": { + "query": "label_values(telemt_build_info, job)", + "refId": "VarJob" + }, + "refresh": 1, + "regexApplyTo": "value", + "type": "query" + }, + { + "allValue": ".*", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "definition": "label_values(telemt_user_connections_total{job=~\"$job\"}, user)", + "includeAll": true, + "label": "User", + "multi": true, + "name": "user", + "options": [], + "query": { + "query": "label_values(telemt_user_connections_total{job=~\"$job\"}, user)", + "refId": "VarUser" + }, + "refresh": 1, + "regexApplyTo": "value", + "type": "query" } ] }, "time": { - "from": "now-6h", + "from": "now-1h", "to": "now" }, "timepicker": {}, @@ -801,4 +4961,4 @@ "title": "Telemt MtProto proxy", "weekStart": "" } -} +} \ No newline at end of file diff --git a/tools/telemt_api.py b/tools/telemt_api.py index 36ba5e1..072ca46 100644 --- a/tools/telemt_api.py +++ b/tools/telemt_api.py @@ -24,7 +24,7 @@ from urllib.request import Request, urlopen # Exceptions # --------------------------------------------------------------------------- -class TememtAPIError(Exception): +class TelemtAPIError(Exception): """Raised when the API returns an error envelope or a transport error.""" def __init__(self, message: str, code: str | None = None, @@ -35,7 +35,7 @@ class TememtAPIError(Exception): self.request_id = request_id def __repr__(self) -> str: - return (f"TememtAPIError(message={str(self)!r}, code={self.code!r}, " + return (f"TelemtAPIError(message={str(self)!r}, code={self.code!r}, " f"http_status={self.http_status}, request_id={self.request_id})") @@ -58,7 +58,7 @@ class APIResponse: # Main client # --------------------------------------------------------------------------- -class TememtAPI: +class TelemtAPI: """ HTTP client for the Telemt Control API. @@ -75,10 +75,10 @@ class TememtAPI: """ def __init__( - self, - base_url: str = "http://127.0.0.1:9091", - auth_header: str | None = None, - timeout: int = 10, + self, + base_url: str = "http://127.0.0.1:9091", + auth_header: str | None = None, + timeout: int = 10, ) -> None: self.base_url = base_url.rstrip("/") self.auth_header = auth_header @@ -98,12 +98,12 @@ class TememtAPI: return h def _request( - self, - method: str, - path: str, - body: dict | None = None, - if_match: str | None = None, - query: dict | None = None, + self, + method: str, + path: str, + body: dict | None = None, + if_match: str | None = None, + query: dict | None = None, ) -> APIResponse: url = self.base_url + path if query: @@ -133,22 +133,22 @@ class TememtAPI: try: payload = json.loads(raw) except Exception: - raise TememtAPIError( + raise TelemtAPIError( str(exc), http_status=exc.code ) from exc err = payload.get("error", {}) - raise TememtAPIError( + raise TelemtAPIError( err.get("message", str(exc)), code=err.get("code"), http_status=exc.code, request_id=payload.get("request_id"), ) from exc except URLError as exc: - raise TememtAPIError(str(exc)) from exc + raise TelemtAPIError(str(exc)) from exc if not payload.get("ok"): err = payload.get("error", {}) - raise TememtAPIError( + raise TelemtAPIError( err.get("message", "unknown error"), code=err.get("code"), request_id=payload.get("request_id"), @@ -298,16 +298,16 @@ class TememtAPI: # ------------------------------------------------------------------ def create_user( - self, - username: str, - *, - secret: str | None = None, - user_ad_tag: str | None = None, - max_tcp_conns: int | None = None, - expiration_rfc3339: str | None = None, - data_quota_bytes: int | None = None, - max_unique_ips: int | None = None, - if_match: str | None = None, + self, + username: str, + *, + secret: str | None = None, + user_ad_tag: str | None = None, + max_tcp_conns: int | None = None, + expiration_rfc3339: str | None = None, + data_quota_bytes: int | None = None, + max_unique_ips: int | None = None, + if_match: str | None = None, ) -> APIResponse: """POST /v1/users — create a new user. @@ -340,16 +340,16 @@ class TememtAPI: return self._post("/v1/users", body=body, if_match=if_match) def patch_user( - self, - username: str, - *, - secret: str | None = None, - user_ad_tag: str | None = None, - max_tcp_conns: int | None = None, - expiration_rfc3339: str | None = None, - data_quota_bytes: int | None = None, - max_unique_ips: int | None = None, - if_match: str | None = None, + self, + username: str, + *, + secret: str | None = None, + user_ad_tag: str | None = None, + max_tcp_conns: int | None = None, + expiration_rfc3339: str | None = None, + data_quota_bytes: int | None = None, + max_unique_ips: int | None = None, + if_match: str | None = None, ) -> APIResponse: """PATCH /v1/users/{username} — partial update; only provided fields change. @@ -385,10 +385,10 @@ class TememtAPI: if_match=if_match) def delete_user( - self, - username: str, - *, - if_match: str | None = None, + self, + username: str, + *, + if_match: str | None = None, ) -> APIResponse: """DELETE /v1/users/{username} — remove user; blocks deletion of last user. @@ -403,11 +403,11 @@ class TememtAPI: # in the route matcher (documented limitation). The method is provided # for completeness and future compatibility. def rotate_secret( - self, - username: str, - *, - secret: str | None = None, - if_match: str | None = None, + self, + username: str, + *, + secret: str | None = None, + if_match: str | None = None, ) -> APIResponse: """POST /v1/users/{username}/rotate-secret — rotate user secret. @@ -533,12 +533,12 @@ EXAMPLES help="Username for user commands") # user create/patch fields - p.add_argument("--secret", default=None) - p.add_argument("--ad-tag", dest="ad_tag", default=None) + p.add_argument("--secret", default=None) + p.add_argument("--ad-tag", dest="ad_tag", default=None) p.add_argument("--max-conns", dest="max_conns", type=int, default=None) - p.add_argument("--expires", default=None) - p.add_argument("--quota", type=int, default=None) - p.add_argument("--max-ips", dest="max_ips", type=int, default=None) + p.add_argument("--expires", default=None) + p.add_argument("--quota", type=int, default=None) + p.add_argument("--max-ips", dest="max_ips", type=int, default=None) # events p.add_argument("--limit", type=int, default=None, @@ -564,10 +564,10 @@ if __name__ == "__main__": sys.exit(0) if cmd == "gen-secret": - print(TememtAPI.generate_secret()) + print(TelemtAPI.generate_secret()) sys.exit(0) - api = TememtAPI(args.url, auth_header=args.auth, timeout=args.timeout) + api = TelemtAPI(args.url, auth_header=args.auth, timeout=args.timeout) try: # -- read endpoints -------------------------------------------------- @@ -690,7 +690,8 @@ if __name__ == "__main__": parser.error("patch command requires ") if not any([args.secret, args.ad_tag, args.max_conns, args.expires, args.quota, args.max_ips]): - parser.error("patch requires at least one field (--secret, --max-conns, --expires, --quota, --max-ips, --ad-tag)") + parser.error( + "patch requires at least one field (--secret, --max-conns, --expires, --quota, --max-ips, --ad-tag)") _print(api.patch_user( args.arg, secret=args.secret, @@ -721,7 +722,7 @@ if __name__ == "__main__": file=sys.stderr) sys.exit(1) - except TememtAPIError as exc: + except TelemtAPIError as exc: print(f"API error [{exc.http_status}] {exc.code}: {exc}", file=sys.stderr) sys.exit(1) except KeyboardInterrupt: