Models
The ML models the sidecar loads and their CPU/GPU residency.
Sidecar model topology
The sidecar is a stateless calc server. It loads the following models:
| Slot | Model | Backend | Residency |
|---|---|---|---|
| STT (default) | NVIDIA Parakeet TDT 0.6B v3 (ONNX via onnx-asr) | onnxruntime, configurable execution provider | GPU (audio.stt.parakeet.execution_provider; auto, cuda, rocm, dml, or cpu) |
| STT (opt-in) | whisper.cpp large-v3-turbo (ggml) | pywhispercpp, backend-accelerated wheel | GPU (Vulkan/CUDA/Metal; CPU if no matching wheel) |
| VAD | Silero VAD v5 (ONNX) | onnxruntime | CPU (CPUExecutionProvider), always-resident |
| Turn — livekit | livekit/turn-detector v0.4.1-intl (ONNX, LlamaForCausalLM) | onnxruntime + HF tokenizer | CPU (CPUExecutionProvider), always-resident |
| Turn — smart | pipecat-ai/smart-turn-v3 (ONNX, Whisper-Tiny encoder + linear head) | onnxruntime | CPU (CPUExecutionProvider), always-resident |
| LLM | configurable GGUF (default: Gemma) | xllamacpp (Vulkan/CUDA) | GPU, unloaded after audio.tts.unload_timeout_s idle |
| TTS | OmniVoice / Piper | xllamacpp / piper binary | GPU (OmniVoice) or CPU (Piper) |
| Text embedder | paraphrase-multilingual-MiniLM-L12-v2 (memory recall/ranking/fact formation + tool arming, one shared instance behind EmbedText) | sentence-transformers (PyTorch) | CPU (default; configurable via memory.embedding_device), preloaded at boot behind READY |
| Speaker | ERes2NetV2 (ONNX) | onnxruntime | CPU, lazy (when speaker.enabled=true), backs EmbedVoice calc verb |
VAD, livekit, and smart_turn always run on CPU (CPUExecutionProvider) and are always-resident — they never occupy a GPU slot and never compete with STT/LLM/TTS for VRAM. STT/LLM/TTS are the GPU-resident slots.
Remote LLM providers (Anthropic and OpenAI-compatible: OpenAI, DeepSeek, Gemini, Ollama, LM Studio, vLLM) never appear in this table — they don't load into the sidecar at all. llm.used_provider selects between the local GGUF slot above and a remote provider; when a remote provider is selected, the Go server's internal/core/provider/apillm backend talks to the API directly over HTTP and the sidecar's LLM slot stays cold.
End-of-speech detection
The end-of-speech statemachine lives in the Go server (internal/core/endpoint/), not in the sidecar. The sidecar exposes ClassifyTurn — a stateless calc RPC that runs livekit and smart_turn ONNX inference and returns raw probabilities plus calibration boundaries. Go owns fusion, hysteresis, the lerp-based silence wait, and the final endpoint decision. The sidecar emits per-frame vad_verdict over the Listen stream; Go's statemachine consumes those verdicts alongside ClassifyTurn results to decide turn end.
Warmup (Prepare)
The Prepare control RPC warms all model slots and streams per-slot readiness back to Go. The turn bundle (VAD + livekit + smart_turn) is always-resident and is not torn down by a GPU-model reload (LLM/TTS swap). Re-calling Prepare on an already-warm slot is a no-op that still reports ready=True.
Swapping the LLM
Configure llm.providers.xllamacpp.model_file (GGUF path or HuggingFace repo+file) and llm.providers.xllamacpp.model_repo. The sidecar self-heals a partial download on next start.
Downloads & caching
Models download automatically on first use and are re-validated on every start — see Self-healing.