On-Device Inference

On-device inference runs the model on the operator's phone or tablet. Once the model files are installed, local model requests can work without a network connection.

Why on-device

Local inference removes the model's dependence on a remote server. It is useful when network coverage is intermittent or the deployment restricts remote processing. Battery, available memory, and device temperature still affect operation.

Keeping inference local reduces where model context is sent. It does not determine the handling requirements for the device or the rest of the deployment.

Choose a model that can call the tools you need within the device's memory and latency limits. You can also configure remote providers and a fallback order when the deployment permits them.

Two runtimes

llama.cpp

Runs as a native HTTP subprocess on the device. The agent connects through its OpenAI-compatible API on localhost.

• Model format: GGUF (quantized, single-file)
• GPU: Vulkan (optional)
• Model support: Supports multiple model families in GGUF format
• Trade-off: Full prompt re-processing on every reasoning iteration

LiteRT-LM

Google's on-device inference SDK. Runs in-process via JNI -- no subprocess, no HTTP overhead.

• Model format: .litertlm (converted from SafeTensors)
• GPU: OpenCL (automatic detection)
• Maturity: Newer, fewer models available
• Session state: Retains context across reasoning iterations, allowing follow-up turns to process new tokens without rebuilding the full prompt. The latency benefit depends on the request and device.

Recommended models

See Tested models for the hardware and flows used in verification:

The "verified" label refers to combinations tested with CUST/OS skill and chat flows. A model that generates text still needs testing for tool names, arguments, and coordinate accuracy.

Current recommendation

Gemma 4 E2B-it on LiteRT-LM.

Backend pick depends on whether you've validated your hardware:

• GPU backend — verified numerically stable on Samsung S26 Ultra (Adreno 840) with our custom 0.11.0-custos LiteRT-LM build, which applies an FP32 activation patch that fixes the multi-digit coordinate rounding the stock OSS build exhibited. Recommended once validated on your specific device.
• CPU backend — the default for initial setup. Test speed and output accuracy on your device before deployment.

Configure as a handheld tier provider:

providers:
  - name: "on-device-gemma4"
    task: chat
    protocol: litert
    model: "gemma-4-E2B-it"
    url: "file:///sdcard/atak/custos/models/gemma-4-E2B-it.litertlm"
    tier: handheld
    taskPriority: 1
    contextSize: 16384
    threads: 4
    properties:
      backend: "cpu"     # "cpu", "gpu", or "npu"

The npu backend (Hexagon / dedicated NPU) is also exposed but not verified on any device yet — treat as experimental.

Performance expectations

Performance depends on the model, prompt, context size, and hardware:

• LiteRT-LM — the first request processes the initial prompt. Follow-up reasoning iterations reuse the KV cache, reducing the amount of prompt processing needed.
• llama.cpp CPU — consistent but slower throughput. Memory bandwidth is the bottleneck on mobile devices. Every ReAct iteration re-processes the full prompt.
• LiteRT GPU — benchmark against CPU on the target device. Verify multi-digit coordinate accuracy as well as speed.

Serialized sessions

LiteRT-LM permits only one active Conversation per Engine at a time. The adapter serializes every inference call through a mutex, which means background tasks (like the skill reranker or embedding rebuild) and foreground chat never run in parallel against the same LiteRT model — they interleave. You'll notice this as a brief pause in the chat stream when a background request fires. It's intentional: collisions produced FAILED_PRECONDITION: A session already exists errors that tripped the router's circuit breaker, so the adapter takes the lock instead.

If you need true parallelism (chat + background reranker running simultaneously), configure them against different providers — e.g. chat on the local Gemma 4 and embeddings on a LAN server.

What is next

• Broader device verification of the GPU backend — the 0.11.0-custos FP32 activation patch should generalize across OpenCL GPUs, but each vendor's driver needs its own coordinate-precision check.
• NPU backend verification on devices with Hexagon / dedicated NPU paths.
• Measure memory use and throughput on additional devices.
• Evaluate smaller models for tool-calling reliability and download size.

See also

• Reference: Providers
• Explanation: Tiers and priority
• Reference: Configuration