telemt/docs/model/MODEL.en.md

Telemt Runtime Model

Scope

This document defines runtime concepts used by the Middle-End (ME) transport pipeline and the orchestration logic around it.

It focuses on:

• ME Pool / Reader / Writer / Refill / Registry
• Adaptive Floor
• Trio-State
• Generation Lifecycle

Core Entities

ME Pool

ME Pool is the runtime orchestrator for all Middle-End writers.

Responsibilities:

• Holds writer inventory by DC/family/endpoint.
• Maintains routing primitives and writer selection policy.
• Tracks generation state (active, warm, draining context).
• Applies runtime policies (floor mode, refill, reconnect, reinit, fallback behavior).
• Exposes readiness gates used by admission logic (for conditional accept/cast behavior).

Non-goals:

• It does not own client protocol decoding.
• It does not own per-client business policy (quotas/limits).

ME Writer

ME Writer is a long-lived ME RPC tunnel bound to one concrete ME endpoint (ip:port), with:

• Outbound command channel (send path).
• Associated reader loop (inbound path).
• Health/degraded flags.
• Contour/state and generation metadata.

A writer is the actual data plane carrier for client sessions once bound.

ME Reader

ME Reader is the inbound parser/dispatcher for one writer:

• Reads/decrypts ME RPC frames.
• Validates sequence/checksum.
• Routes payloads to client-connection channels via Registry.
• Emits close/ack/data events and updates telemetry.

Design intent:

• Reader must stay non-blocking as much as possible.
• Backpressure on a single client route must not stall the whole writer stream.

Refill

Refill is the recovery mechanism that restores writer coverage when capacity drops:

• Per-endpoint restore (same endpoint first).
• Per-DC restore to satisfy required floor.
• Optional outage-mode/shadow behavior for fragile single-endpoint DCs.

Refill works asynchronously and should not block hot routing paths.

Registry

Registry is the routing index between ME and client sessions:

• conn_id -> client response channel
• conn_id <-> writer_id binding map
• writer activity snapshots and idle tracking

Main invariants:

• A conn_id routes to at most one active response channel.
• Writer loss triggers safe unbind/cleanup and close propagation.
• Registry state is the source of truth for active ME-bound session mapping.

Adaptive Floor

What it is

Adaptive Floor is a runtime policy that changes target writer count per DC based on observed activity, instead of always holding static peak floor.

Why it exists

Goals:

• Reduce idle writer churn under low traffic.
• Keep enough warm capacity to avoid client-visible stalls on burst recovery.
• Limit needless reconnect storms on unstable endpoints.

Behavioral model

• Under activity: floor converges toward configured static requirement.
• Under prolonged idle: floor can shrink to a safe minimum.
• Recovery/grace windows prevent aggressive oscillation.

Safety constraints

• Never violate minimal survivability floor for a DC group.
• Refill must still restore quickly on demand.
• Floor adaptation must not force-drop already bound healthy sessions.

Trio-State

Trio-State is writer contouring:

• Warm
• Active
• Draining

State semantics

• Warm: connected and validated, not primary for new binds.
• Active: preferred for new binds and normal traffic.
• Draining: no new regular binds; existing sessions continue until graceful retirement rules apply.

Transition intent

• Warm -> Active: when coverage/readiness conditions are satisfied.
• Active -> Draining: on generation swap, endpoint replacement, or controlled retirement.
• Draining -> removed: after drain TTL/force-close policy (or when naturally empty).

This separation reduces SPOF and keeps cutovers predictable.

Generation Lifecycle

Generation isolates pool epochs during reinit/reconfiguration.

Lifecycle phases

1. Bootstrap: initial writers are established.
2. Warmup: next generation writers are created and validated.
3. Activation: generation promoted to active when coverage gate passes.
4. Drain: previous generation becomes draining, existing sessions are allowed to finish.
5. Retire: old generation writers are removed after graceful rules.

Operational guarantees

• No partial generation activation without minimum coverage.
• Existing healthy client sessions should not be dropped just because a new generation appears.
• Draining generation exists to absorb in-flight traffic during swap.

Readiness and admission

Pool readiness is not equivalent to “all endpoints fully saturated”. Typical gating strategy:

• Open admission when per-DC minimal alive coverage exists.
• Continue background saturation for multi-endpoint DCs.

This keeps startup latency low while preserving eventual full capacity.

Interactions Between Concepts

• Generation defines pool epochs.
• Trio-State defines per-writer role inside/around those epochs.
• Adaptive Floor defines how much capacity should be maintained right now.
• Refill is the actuator that closes the gap between desired and current capacity.
• Registry keeps per-session routing correctness while all of the above changes over time.

Architectural Approach

Layered Design

The runtime is intentionally split into two planes:

• Control Plane: decides desired topology and policy (floor, generation swap, refill, fallback).
• Data Plane: executes packet/session transport (reader, writer, routing, acks, close propagation).

Architectural rule:

• Control Plane may change writer inventory and policy.
• Data Plane must remain stable and low-latency while those changes happen.

Ownership Model

Ownership is centered around explicit state domains:

• MePool owns writer lifecycle and policy state.
• Registry owns per-connection routing bindings.
• Writer task owns outbound ME socket send progression.
• Reader task owns inbound ME socket parsing and event dispatch.

This prevents accidental cross-layer mutation and keeps invariants local.

Control Plane Responsibilities

Control Plane is event-driven and policy-driven:

• Startup initialization and readiness gates.
• Runtime reinit (periodic or config-triggered).
• Coverage checks per DC/family/endpoint group.
• Floor enforcement (static/adaptive).
• Refill scheduling and retry orchestration.
• Generation transition (warm -> active, previous active -> draining).

Control Plane must prioritize determinism over short-term aggressiveness.

Data Plane Responsibilities

Data Plane is throughput-first and allocation-sensitive:

• Session bind to writer.
• Per-frame parsing/validation and dispatch.
• Ack and close signal propagation.
• Route drop behavior under missing connection or closed channel.
• Minimal critical logging in hot path.

Data Plane should avoid waiting on operations that are not strictly required for frame correctness.

Concurrency and Synchronization

Concurrency Principles

• Per-writer isolation: each writer has independent send/read task loops.
• Per-connection isolation: client channel state is scoped by conn_id.
• Asynchronous recovery: refill/reconnect runs outside the packet hot path.

Synchronization Strategy

• Shared maps use fine-grained, short-lived locking.
• Read-mostly paths avoid broad write-lock windows.
• Backpressure decisions are localized at route/channel boundary.

Design target:

• A slow consumer should degrade only itself (or its route), not global writer progress.

Cancellation and Shutdown

Writer and reader loops are cancellation-aware:

• explicit cancel token / close command support;
• safe unbind and cleanup via registry;
• deterministic order: stop admission -> drain/close -> release resources.

Consistency Model

Session Consistency

For one conn_id:

• exactly one active route target at a time;
• close and unbind must be idempotent;
• writer loss must not leave dangling bindings.

Generation Consistency

Generational consistency guarantees:

• New generation is not promoted before minimum coverage gate.
• Previous generation remains available in draining state during handover.
• Forced retirement is policy-bound (drain ttl, optional force-close), not immediate.

Policy Consistency

Policy changes (adaptive/static floor, fallback mode, retries) should apply without violating established active-session routing invariants.

Backpressure and Flow Control

Route-Level Backpressure

Route channels are bounded by design. When pressure increases:

• short burst absorption is allowed;
• prolonged congestion triggers controlled drop semantics;
• drop accounting is explicit via metrics/counters.

Reader Non-Blocking Priority

Inbound ME reader path should never be serialized behind one congested client route. Practical implication:

• prefer non-blocking route attempt in the parser loop;
• move heavy recovery to async side paths.

Failure Domain Strategy

Endpoint-Level Failure

Failure of one endpoint should trigger endpoint-scoped recovery first:

• same endpoint reconnect;
• endpoint replacement within same DC group if applicable.

DC-Level Degradation

If a DC group cannot satisfy floor:

• keep service via remaining coverage if policy allows;
• continue asynchronous refill saturation in background.

Whole-Pool Readiness Loss

If no sufficient ME coverage exists:

• admission gate can hold new accepts (conditional policy);
• existing sessions should continue when their path remains healthy.

Performance Architecture Notes

Hotpath Discipline

Allowed in hotpath:

• fixed-size parsing and cheap validation;
• bounded channel operations;
• precomputed or low-allocation access patterns.

Avoid in hotpath:

• repeated expensive decoding;
• broad locks with awaits inside critical sections;
• verbose high-frequency logging.

Throughput Stability Over Peak Spikes

Architecture prefers stable throughput and predictable latency over short peak gains that increase churn or long-tail reconnect times.

Evolution and Extension Rules

To evolve this model safely:

• Add new policy knobs in Control Plane first.
• Keep Data Plane contracts stable (conn_id, route semantics, close semantics).
• Validate generation and registry invariants before enabling by default.
• Introduce new retry/recovery strategies behind explicit config.

Failure and Recovery Notes

• Single-endpoint DC failure is a normal degraded mode case; policy should prioritize fast reconnect and optional shadow/probing strategies.
• Idle close by peer should be treated as expected when upstream enforces idle timeout.
• Reconnect backoff must protect against synchronized churn while still allowing fast first retries.
• Fallback (ME -> direct DC) is a policy switch, not a transport bug by itself.

Terminology Summary

• Coverage: enough live writers to satisfy per-DC acceptance policy.
• Floor: target minimum writer count policy.
• Churn: frequent writer reconnect/remove cycles.
• Hotpath: per-packet/per-connection data path where extra waits/allocations are expensive.