telemt/docs/Architecture/Model/MODEL.en.md

Telemt Runtime Model

Scope

This document defines runtime concepts used by the Middle-End (ME) transport pipeline and the orchestration logic around it.

It focuses on:

• ME Pool / Reader / Writer / Refill / Registry
• Adaptive Floor
• Trio-State
• Generation Lifecycle

Core Entities

ME Pool

ME Pool is the runtime orchestrator for all Middle-End writers.

Responsibilities:

• Holds writer inventory by DC/family/endpoint.
• Maintains routing primitives and writer selection policy.
• Tracks generation state (active, warm, draining context).
• Applies runtime policies (floor mode, refill, reconnect, reinit, fallback behavior).
• Exposes readiness gates used by admission logic (for conditional accept/cast behavior).

Non-goals:

• It does not own client protocol decoding.
• It does not own per-client business policy (quotas/limits).

ME Writer

ME Writer is a long-lived ME RPC tunnel bound to one concrete ME endpoint (ip:port), with:

• Outbound command channel (send path).
• Associated reader loop (inbound path).
• Health/degraded flags.
• Contour/state and generation metadata.

A writer is the actual data plane carrier for client sessions once bound.

ME Reader

ME Reader is the inbound parser/dispatcher for one writer:

• Reads/decrypts ME RPC frames.
• Validates sequence/checksum.
• Routes payloads to client-connection channels via Registry.
• Emits close/ack/data events and updates telemetry.

Design intent:

• Reader must stay non-blocking as much as possible.
• Backpressure on a single client route must not stall the whole writer stream.

Refill

Refill is the recovery mechanism that restores writer coverage when capacity drops:

• Per-endpoint restore (same endpoint first).
• Per-DC restore to satisfy required floor.
• Optional outage-mode/shadow behavior for fragile single-endpoint DCs.

Refill works asynchronously and should not block hot routing paths.

Registry

Registry is the routing index between ME and client sessions:

• conn_id -> client response channel
• conn_id <-> writer_id binding map
• writer send routes and their replacement state
• writer activity snapshots and idle tracking

Main invariants:

• A conn_id routes to at most one active response channel.
• Writer loss triggers safe unbind/cleanup and close propagation.
• Registry state is the source of truth for active ME-bound session mapping.
• The registry binding lock linearizes client binds, writer publication, and the transition that closes a replacement victim to new binds.

Adaptive Floor

What it is

Adaptive Floor is a runtime policy that changes target writer count per DC based on observed activity, instead of always holding static peak floor.

Why it exists

Goals:

• Reduce idle writer churn under low traffic.
• Keep enough warm capacity to avoid client-visible stalls on burst recovery.
• Limit needless reconnect storms on unstable endpoints.

Behavioral model

• Under activity: floor converges toward configured static requirement.
• Under prolonged idle: floor can shrink to a safe minimum.
• Recovery/grace windows prevent aggressive oscillation.

Safety constraints

• Never violate minimal survivability floor for a DC group.
• Refill must still restore quickly on demand.
• Floor adaptation must not force-drop already bound healthy sessions.

Trio-State

Trio-State is writer contouring:

• Warm
• Active
• Draining

State semantics

• Warm: connected and validated, not primary for new binds.
• Active: preferred for new binds and normal traffic.
• Draining: no new regular binds; existing sessions continue until graceful retirement rules apply.

Transition intent

• Warm -> Active: when coverage/readiness conditions are satisfied.
• Active -> Draining: on generation swap, endpoint replacement, or controlled retirement.
• Draining -> removed: when naturally empty, at the effective force-close deadline, or through threshold/control-path eviction. me_pool_drain_ttl_secs is a warning threshold and force-close lower bound, not a removal deadline by itself.

Writer replacement is a separate registry-local lifecycle: Open -> Preparing -> Retiring. Preparing excludes duplicate replacement work but intentionally permits new client binds. Commit revalidates the victim under the binding lock; Retiring rejects new binds. Dropping an uncommitted reservation restores Open, while the writer's contour remains independently Warm, Active, or Draining.

This separation reduces SPOF and keeps cutovers predictable.

Generation Lifecycle

Generation isolates pool epochs during reinit/reconfiguration.

Lifecycle phases

1. Bootstrap: initial writers are established.
2. Warmup: next generation writers are created and validated.
3. Activation: generation is promoted atomically when the configured coverage ratio and stale-binding policy pass commit-time revalidation.
4. Drain: policy-eligible old writers remain as bounded stale fallback; covered old writers become ineligible and enter retirement during the commit, then may close immediately after it returns.
5. Retire: draining writers are removed when empty or by force-close, threshold, or explicit control policy.

Operational guarantees

• Activation is atomic, but the committed topology may still have missing DC-family groups when me_pool_min_fresh_ratio passes and me_bind_stale_mode permits bounded stale fallback. Mode never rejects any missing group.
• Generation handover is policy-bound, not universally zero-drop: covered old writers may be retired and their bound sessions closed immediately after commit, while only selected stale writers remain available for uncovered groups.
• A pending generation owns only writers accepted for its generation and current endpoint map; stale tasks cannot publish into a newer generation.
• A pending generation is keyed by desired-map hash and endpoint revision and may be reused for up to 1800 seconds before expiring.
• Writer replacement prepares a successor; under one binding guard, commit first moves the predecessor to Retiring and then registers the successor before releasing the guard. Failed or cancelled preparation before that boundary preserves the predecessor and releases the reservation.
• Pool-state telemetry exposes pending writer count and deficit, missing DC-family groups, map currency, orphan warm writers, and replacement preparing/retiring counts.

Readiness and admission

Pool readiness is not equivalent to “all endpoints fully saturated”. Typical gating strategy:

• Open admission when per-DC minimal alive coverage exists.
• Continue background saturation for multi-endpoint DCs.

This keeps startup latency low while preserving eventual full capacity.

Interactions Between Concepts

• Generation defines pool epochs.
• Trio-State defines per-writer role inside/around those epochs.
• Adaptive Floor defines how much capacity should be maintained right now.
• Refill is the actuator that closes the gap between desired and current capacity.
• Registry keeps per-session routing correctness while all of the above changes over time.

Architectural Approach

Layered Design

The runtime is intentionally split into two planes:

• Control Plane: decides desired topology and policy (floor, generation swap, refill, fallback).
• Data Plane: executes packet/session transport (reader, writer, routing, acks, close propagation).

Architectural rule:

• Control Plane may change writer inventory and policy.
• Data Plane must remain stable and low-latency while those changes happen.

Ownership Model

Ownership is centered around explicit state domains:

• MePool owns writer inventory, contour lifecycle, and runtime policy state.
• The reinit coordinator owns active/pending generation authority keyed by map hash and endpoint revision.
• Registry owns per-connection routing bindings, writer send routes, and writer replacement state.
• Writer task owns outbound ME socket send progression.
• Reader task owns inbound ME socket parsing and event dispatch.

This prevents accidental cross-layer mutation and keeps invariants local.

Control Plane Responsibilities

Control Plane is event-driven and policy-driven:

• Startup initialization and readiness gates.
• Runtime reinit (periodic or config-triggered).
• Coverage checks per DC/family/endpoint group.
• Floor enforcement (static/adaptive).
• Refill scheduling and retry orchestration.
• Generation transition (warm -> active, previous active -> draining).

Control Plane must prioritize determinism over short-term aggressiveness.

Data Plane Responsibilities

Data Plane is throughput-first and allocation-sensitive:

• Session bind to writer.
• Per-frame parsing/validation and dispatch.
• Ack and close signal propagation.
• Route drop behavior under missing connection or closed channel.
• Minimal critical logging in hot path.

Data Plane should avoid waiting on operations that are not strictly required for frame correctness.

Concurrency and Synchronization

Concurrency Principles

• Per-writer isolation: each writer has independent send/read task loops.
• Per-connection isolation: client channel state is scoped by conn_id.
• Asynchronous recovery: refill/reconnect runs outside the packet hot path.

Synchronization Strategy

• Shared maps use fine-grained, short-lived locking.
• Read-mostly paths avoid broad write-lock windows.
• Backpressure decisions are localized at route/channel boundary.
• Generation and replacement commits use the lock order writers -> registry binding -> reinit coordinator.
• After acquiring the registry publication guard, a commit has no cancellation point before publication and retirement state are made consistent.

Design target:

• A slow consumer should degrade only itself (or its route), not global writer progress.

Cancellation and Shutdown

Writer and reader loops are cancellation-aware:

• explicit cancel token / close command support;
• safe unbind and cleanup via registry;
• RAII replacement reservations restore Preparing to Open when preparation is cancelled before commit;
• deterministic order: stop admission -> drain/close -> release resources.

Consistency Model

Session Consistency

For one conn_id:

• exactly one active route target at a time;
• close and unbind must be idempotent;
• writer loss must not leave dangling bindings.

Generation Consistency

Generational consistency guarantees:

• Commit revalidates generation, desired-map hash, endpoint revision, and fresh coverage while holding the publication barriers.
• Promotion requires me_pool_min_fresh_ratio; missing DC-family groups additionally require a stale-binding mode other than never.
• Previous-generation writers are retained only where the selected stale-fallback policy requires them. Covered writers become ineligible at commit and may close immediately afterward.
• Draining writers are removed when empty, at the effective force-close deadline (0 first selects the 300-second safety fallback, then the drain TTL remains a lower bound), or by threshold/control-path eviction; drain TTL alone only triggers warnings.

Policy Consistency

Policy changes (adaptive/static floor, fallback mode, retries) should apply without violating established active-session routing invariants.

Backpressure and Flow Control

Route-Level Backpressure

Route channels are bounded by design. When pressure increases:

• short burst absorption is allowed;
• prolonged congestion triggers controlled drop semantics;
• drop accounting is explicit via metrics/counters.

Reader Non-Blocking Priority

Inbound ME reader path should never be serialized behind one congested client route. Practical implication:

• prefer non-blocking route attempt in the parser loop;
• move heavy recovery to async side paths.

Failure Domain Strategy

Endpoint-Level Failure

Failure of one endpoint should trigger endpoint-scoped recovery first:

• same endpoint reconnect;
• endpoint replacement within same DC group if applicable.

DC-Level Degradation

If a DC group cannot satisfy floor:

• keep service via remaining coverage if policy allows;
• continue asynchronous refill saturation in background.

Whole-Pool Readiness Loss

If no sufficient ME coverage exists:

• admission gate can hold new accepts (conditional policy);
• existing sessions should continue when their path remains healthy.

Performance Architecture Notes

Hotpath Discipline

Allowed in hotpath:

• fixed-size parsing and cheap validation;
• bounded channel operations;
• precomputed or low-allocation access patterns.

Avoid in hotpath:

• repeated expensive decoding;
• broad locks with awaits inside critical sections;
• verbose high-frequency logging.

Throughput Stability Over Peak Spikes

Architecture prefers stable throughput and predictable latency over short peak gains that increase churn or long-tail reconnect times.

Evolution and Extension Rules

To evolve this model safely:

• Add new policy knobs in Control Plane first.
• Keep Data Plane contracts stable (conn_id, route semantics, close semantics).
• Validate generation and registry invariants before enabling by default.
• Introduce new retry/recovery strategies behind explicit config.

Failure and Recovery Notes

• Single-endpoint DC failure is a normal degraded mode case; policy should prioritize fast reconnect and optional shadow/probing strategies.
• Idle close by peer should be treated as expected when upstream enforces idle timeout.
• Reconnect backoff must protect against synchronized churn while still allowing fast first retries.
• Fallback (ME -> direct DC) is a policy switch, not a transport bug by itself.

Terminology Summary

• Coverage: enough live writers to satisfy per-DC acceptance policy.
• Floor: target minimum writer count policy.
• Churn: frequent writer reconnect/remove cycles.
• Hotpath: per-packet/per-connection data path where extra waits/allocations are expensive.