Add an On-Device Model

CUST/OS can run LLMs, embedding models, speech-to-text, and vision detection entirely on-device with no network connectivity. This guide shows how to configure each kind.

How on-device models work

For subprocess backends such as llama.cpp and Whisper, a file:// provider starts a native server on the configured port. CUST/OS connects to http://127.0.0.1:<port> for inference. LiteRT-LM loads the model in process instead. Both approaches use model files stored on the device.

Add an LLM

1. Get the model

GGUF models are the standard format. Download one from HuggingFace or your model provider:

adb push qwen2.5-7b-instruct-q4_k_m.gguf /sdcard/atak/custos/models/

2. Add a provider in custos.yaml

providers:
  - name: "qwen-on-device"
    task: "chat"
    protocol: "openai"
    runtime: "llama.cpp"
    url: "file:///sdcard/atak/custos/models/qwen2.5-7b-instruct-q4_k_m.gguf"
    port: 8411
    model: "qwen2.5-7b"
    contextSize: 8192
    threads: 4
    chatTemplatePath: "/sdcard/atak/custos/models/qwen3-tool-calling.jinja"
    requestTimeoutMs: 120000
    taskPriority: 100
    tier: "handheld"

3. Save and verify

Save custos.yaml and wait for the model to load. Check the Status panel; if the provider stays offline, tap its row to read the error.

Chat templates

The on-device LLM server uses jinja templates to render chat messages and tool-call instructions. Without a tool-calling template the model cannot emit tool calls in a parseable format. Templates ship in /sdcard/atak/custos/models/:

• qwen3-tool-calling.jinja -- for Qwen models
• gemma3-tool-calling.jinja -- for Gemma models

If you bring a different model family, drop in its tool-calling template and point chatTemplatePath at it.

Add an embedding model

Embeddings use the same configuration pattern with task: embedding:

  - name: "local-embed"
    task: "embedding"
    protocol: "openai"
    runtime: "llama.cpp"
    url: "file:///sdcard/atak/custos/models/nomic-embed-text-v1.5.Q8_0.gguf"
    port: 8414
    model: "nomic-embed-text-v1.5"
    contextSize: 512
    threads: 4
    taskPriority: 10

Embeddings power skill selection (semantic matching against the operator's message) and the vector store used by the custos.rag skill. Without an embedding provider, skill selection falls back to keyword-only matching -- still functional, just less precise.

Add a speech-to-text model (Whisper)

  - name: "on-device-whisper"
    task: "transcription"
    protocol: "openai"
    runtime: "whisper.cpp"
    url: "file:///sdcard/atak/custos/models/ggml-tiny.bin"
    port: 8412
    model: "whisper-tiny"
    taskPriority: 100
    tier: "handheld"

Whisper models use GGML format (.bin). The example uses ggml-tiny.bin (77 MB). Test transcription with the speech and background noise you expect in use. Once the provider is online, it can transcribe PTT recordings on the device.

Add a vision detection model (ONNX)

  - name: "yolo-detection"
    task: "detection"
    protocol: "vision"
    runtime: "onnxruntime"
    url: "file:///sdcard/atak/custos/models/yolo11m.onnx"
    port: 8413
    model: "yolo11m"
    threads: 4
    confidence: 0.15
    inputSize: 640
    taskPriority: 1
    tier: "handheld"

Vision detection is used by skills like custos.detect_buildings. Custom labels can be specified in the properties: map.

Add an LLM (LiteRT-LM, Gemma-4-family)

LiteRT-LM is Google's on-device inference runtime. It runs in process and retains the KV cache across reasoning iterations to reduce repeated prompt processing.

Model format

LiteRT-LM requires .litertlm files. A GGUF file cannot be used with this backend. Follow the starter kit's model download instructions for a compatible file.

Setup (verified: Gemma 4 E2B-it)

adb push gemma-4-E2B-it.litertlm /sdcard/atak/custos/models/

providers:
  - name: "on-device-gemma4"
    task: "chat"
    protocol: "litert"
    url: "file:///sdcard/atak/custos/models/gemma-4-E2B-it.litertlm"
    model: "gemma-4-E2B-it"
    tier: "handheld"
    taskPriority: 1
    contextSize: 16384
    threads: 4
    properties:
      backend: "cpu"     # "cpu" (default), "gpu", or "npu"

Model size

• Gemma 4 E2B-it — tested at contextSize: 16384 on the 16 GB reference device. Check available memory on other devices.
• Gemma 4 E4B-it — exceeded memory at 16K context on the reference device. Start with 4096 or 8192 and test memory use, or use E2B-it. See Tested models.

Backend selection

• cpu (default) — FP32 inference. Use this for initial setup, then test output and speed on your device.
• gpu — OpenCL. The custom 0.11.0-custos runtime has been tested for coordinate precision on Samsung S26 Ultra (Adreno 840). Validate output accuracy and performance before using it on other hardware.
• npu — Hexagon / dedicated NPU path. Not verified on any device yet; treat as experimental.

Model format summary

Tuning for handheld devices

Handheld Android devices are RAM-constrained. Recommendations:

• Try quantized models -- for GGUF, Q4_K_M is a starting point for testing memory use and tool-calling accuracy.
• Tune threads -- start with the number of performance cores on the device, then compare throughput.
• Adjust contextSize -- a smaller context can reduce memory use, but must still fit your prompts, skill definitions, and conversation history.
• Pre-warm before mission -- the first inference call after startup is slow because the model is loading into RAM. Trigger it during pre-mission checks.
• Set taskPriority for your deployment -- use a lower number to prefer the local model. Add remote fallback only if your deployment permits it.

Vulkan GPU offload (llama.cpp)

If your device has a Vulkan-capable GPU (Snapdragon 8 Gen 2+ or newer), you can offload layers:

providers:
  - name: "qwen-vulkan"
    task: "chat"
    protocol: "openai"
    runtime: "llama.cpp"
    url: "file:///sdcard/atak/custos/models/qwen2.5-7b-instruct-q4_k_m.gguf"
    port: 8411
    model: "qwen2.5-7b"
    contextSize: 4096
    threads: 4
    properties:
      gpu-layers: "99"
    taskPriority: 100
    tier: "handheld"

CPU backend is recommended for reliability unless you have benchmarked your specific device and confirmed GPU offload provides a meaningful improvement.

Verify

Check each local provider in the Status panel after loading. If one stays offline, open its details and check the model path, format, and available memory.

Related

• How-to: Add a cloud provider
• Tutorial: Add a provider