Inference Routing

CUST/OS can use separate providers for chat, embeddings, speech, and vision. The inference router selects a provider for each request using the configured priority, security mode, classification level, and provider health.

The problem

Providers may run in several locations, each with different connection requirements:

Handheld -- a model on the operator's device, limited by its memory, battery, and processing capacity.
Pack -- a model on compute carried by the operator and connected by wire.
Mobile -- a model on a squad-carried device, reachable over the local network.
Mounted -- a model on vehicle compute, usable while the connection is available.
Command post -- a workstation reached through the tactical mesh. CoT delegation is in development and has not been validated between live devices.
Cloud -- an internet-hosted model that requires network access and valid provider configuration.

Configure a fallback provider if work needs to continue after a connection or server fails. The router can only use providers that are available and permitted by your settings.

Per-task routing

The router maintains a separate provider list for each task type: chat, embedding, transcription, text-to-speech, and vision. A request uses the list for its task.

Within each task, providers are sorted by priority (lower number wins). On every inference call:

Walk the sorted provider list.
Skip providers that are unhealthy or blocked by the current security mode.
Try the call.
On success, return the result.
On failure, skip to the next provider.
If every provider fails, report the error.

Set provider priorities in the configuration. Lower numbers are tried first after filtering.

Health monitoring

Providers are checked periodically (every few seconds by default). Each check verifies that the provider's endpoint is reachable and responsive. The result is one of:

Online -- healthy, accepting requests.
Slow -- responding, but with degraded latency.
Offline -- unreachable or erroring.

The operator can see provider health at a glance in the status panel. Health states update in real time.

Automatic failover

When a provider fails repeatedly, the router temporarily removes it from consideration and falls back to the next provider in the priority list. After a cooldown period, the failed provider is retried. If it succeeds, it resumes service. If it fails again, it is set aside for another cooldown.

A fallback can keep a request running after a provider fails. Retrying may add delay, and the replacement model may produce different results. If no eligible provider succeeds, the request reports an error.

The router only triggers failover on infrastructure failures (timeouts, connection refused, server errors). Content failures (the model says "I don't know") are valid responses that do not affect provider health.

Tier-based filtering

Providers are grouped by tier based on their physical and trust environment:

Tier	What lives here
`handheld`	On-device models. No network involved.
`pack`	Compute carried by the operator, with a wired inference connection.
`mobile`	Squad-carried companion compute.
`mounted`	Vehicle compute platform.
`command-post`	Mesh-reachable workstations.
`cloud`	Internet-hosted providers.

Security modes filter which tiers are allowed:

Normal -- all tiers active.
No-cloud -- cloud providers blocked.
Field Only -- handheld, pack, mobile, and mounted. No command post or cloud.
Squad Only -- handheld, pack, and mobile only.
EMCON -- handheld and pack providers only.
Standalone -- handheld only. No external compute trusted -- not even wired peripherals.

The router applies the selected mode when choosing a provider. These settings filter inference traffic; they do not turn off radios or control all network activity on the device.

For the full reasoning behind tiers, see Tiers and priority.

Classification boundary

Before selecting a provider, the router compares its declared classification level with the configured operator level. Providers below that level are excluded. This check relies on the deployment's configuration; a provider label does not establish authorization to process the data.

Streaming

For chat, the router streams tokens as the model generates them. The operator sees the response being built in real time, not at the end. If a provider doesn't support streaming, the response is delivered as a single block -- the agent handles both modes transparently.

On-device models

Local model providers initialize after plugin startup. Subprocess backends launch local servers; LiteRT-LM loads in process. A provider appears online after its model or endpoint is ready. Loading time depends on the device and model.

On-device models participate in the same routing, health monitoring, and failover infrastructure as remote providers. From the router's perspective, they are just another provider with a different tier.

LiteRT-LM providers in particular serialize every request through a per-engine mutex — only one conversation runs at a time against a given LiteRT model. That means chat and background reranking interleave rather than run in parallel. See on-device inference for detail and mitigations.

What this gets you

You can configure different providers for each task, choose their fallback order, and restrict selection by tier and classification. Test the configuration with links disconnected as well as connected, so you know which tasks remain available when a provider drops out.