Audio pipeline gotchas

• Loaded ≠ warm — every streaming model runs a throwaway inference at load. onnxruntime (and whisper.cpp/GPU graphs) pay first-run costs on the first session.run, not at session open; paying them inside the first live turn blocks the per-session worker long enough to starve the Go endpoint of vad_verdicts and no-speech-timeout the turn (#529). STT/TTS providers expose warmup() (called by ModelThread after load(), before readiness — so READY implies warm); the turn bundle (Silero VAD, LiveKit, Smart Turn) is warmed by _load_turn_providers, which drops a provider whose warmup inference fails (it could only ever return silence). Any new streaming model must warm at load, never on its first live inference.
• The embedder is part of the READY contract. The one shared multilingual embedder (EmbedText, serving both memory and tool arming) preloads on a background thread at servicer init, serialised behind the shared MODEL_BUILD_LOCK with the pipeline loads, and IsReady gates on that preload. Go calls EmbedText on the activation path before the START event opens capture (context ranking, recall) and again at STT-final (tool arming), so a cold embedder behind READY stalls the whole Listen path — the model once built lazily on the first EmbedText RPC and held the build lock through a minutes-long HF download while the first turn's START waited. The EmbedText handler never builds: a not-yet-warm (or failed-load) embedder answers the soft not-ready response (empty embeddings, dim=0) that Go treats as a value. READY implies warm for every model a turn touches; after READY only an explicit registry hot-swap builds torch weights.
• The Listen path never waits on the STT worker (vad_verdicts must always flow). The session worker that processes audio chunks is the thread that delivers per-frame verdicts to the Go endpoint's end-of-speech detector, and the STT ModelThread can be wedged for minutes behind a model build (a hot-swap load or watchdog reload queuing on MODEL_BUILD_LOCK while some other model — e.g. an embedder — builds). So _PipelineSTTStream keeps the live-audio calls non-blocking: open/feed/feed_vad are fire-and-forget work items (FIFO order is the synchronisation), and partial() is asynchronous with at most one inference in flight — it returns the previously completed text. On top of that the worker skips the partial entirely while more input is queued (_handle_audio_chunk's backlog gate). Only the turn-boundary calls (finalize/discard) block. Partials are throttled live-transcript UX, never load-bearing.
• Silero VAD v5 needs state/stateN (not h/c/hn/cn) AND 576-sample input (512 + 64 context). Wrong interface causes silent 0% speech detection.
• livekit/turn-detector is LlamaForCausalLM, not a classifier. Use ONNX onnx/model_q8.onnx at revision v0.4.1-intl via onnxruntime.
• pipecat-ai/smart-turn-v3 is a Whisper-Tiny encoder + linear head exported to ONNX (sigmoid in-graph). Runs CPU-only via onnxruntime (~12 ms), no torch. Consumes Whisper log-mel input_features[1,80,800], NOT raw PCM — the mel frontend is hand-ported in numpy (validated against WhisperFeatureExtractor by test_smart_turn_mel.py). Output is a completion probability; do NOT apply sigmoid again. The TurnController fuses it with LiveKit via min-fusion dynamic endpointing (confidence sets the silence wait), not OR-fire.
• Neither Parakeet nor whisper.cpp has a streaming decoder — both run stateless offline passes. Long utterances are handled by VAD-segmented incremental finalize (providers/stt_segmenter.py): a silence run of ~416 ms closes a segment, one offline pass per segment, transcripts concatenate. No fixed buffer cap — nothing is dropped, memory and per-pass latency stay bounded.
• Silero VAD jitters on single frames (breath, keyboard, leftover TTS echo). TurnController.on_speech_frame applies _SPEECH_HANGOVER_FRAMES=6 hysteresis before VAD onset is declared.

STT provider

ParakeetSTT (NVIDIA Parakeet TDT 0.6B v3, consumed as the community ONNX export via onnx-asr) is the default STT engine. onnx-asr depends only on numpy + onnxruntime — no torch in the STT path — and spans CUDA / ROCm / DirectML / CPU from a single API. The onnxruntime execution provider is picked at load() time inside the provider (audio.stt.parakeet quantization + execution-provider knobs). StubSTT remains for tests.

Parakeet has no streaming decoder, so it runs as a VAD-segmented incremental stream (SegmentedSTTStream in providers/stt_segmenter.py): the servicer tees its per-frame Silero verdicts into the stream; a silence run of ~416 ms closes the open segment, which gets one offline pass whose text is accumulated, and finalize transcribes only the residual segment — so arbitrarily long utterances transcribe completely. A speaker who never pauses (or a caller feeding no verdicts, like the one-shot Transcribe RPC) is bounded by a forced cut at 15 s on the segment's lowest-energy frame. Partials run over the last 5 s of the open segment, prefixed with the accumulated segment texts. Silence stretches > ~500 ms are trimmed from each pass (verdict-driven mid-turn, shared-Silero fallback at finalize), and a shared youtube-subtitle hallucination blocklist drops boilerplate from every pass.

If ParakeetSTT fails to load (missing onnx-asr / onnxruntime wheel, model download failure, GPU OOM), the STT thread enters degraded state and stays there until the next sidecar restart. A partial/corrupt model cache self-heals: a missing weight file forces a fresh snapshot_download and one retry.

whisper.cpp (opt-in second engine)

WhisperCppSTT (whisper.cpp via pywhispercpp) is an opt-in alternative, selected with audio.stt.engine = STT_ENGINE_WHISPER_CPP; Parakeet stays the default. It shares Parakeet's segmented stream (SegmentedSTTStream), the silence trim, and the hallucination blocklist — only the decoder differs. The ggml model (audio.stt.whisper_cpp.model, default large-v3-turbo) is resolved against ggerganov/whisper.cpp on Hugging Face and self-heals a truncated cache; decode is greedy (pywhispercpp's params struct exposes no flat beam_size).

Upstream pywhispercpp on PyPI is CPU-only, so the engine needs a backend-accelerated wheel (Vulkan on ROCm/AMD, CUDA on NVIDIA, Metal on Apple). internal/sidecar/binprep/whispercpp fetches the host-matching wheel from the solace-assistant/whispercpp-builds release matrix (pinned tag) into the venv, and uv sync --inexact keeps it across syncs — it is deliberately not a uv.lock dependency. On a host with no matching wheel the engine is simply unavailable and load() raises a clear error; the host stays on Parakeet. There is no runtime source-build fallback.

AEC is a client-side concern

Running AEC server-side would require streaming a per-satellite reference signal back over the wire and adds latency without solving the problem cleanly. Each satellite cancels its own speaker echo locally with access to its real playback latency. The Android client uses WebRTC AEC3; ESP32 and similar resource-bound satellites are expected to use platform AEC where available.

Wakeword greeting gate

When a wake identifies a known user with a configured greeting (users.<id>.greeting, level high/medium), the server speaks the cached greeting wav before listening — Alexa-style turn-taking ("Jarvis" → "Sie wünschen?" → command). This is the one place the control plane deliberately gates the data plane.

Mechanics (internal/transport/listener/greeting.go, wake_arbiter.go → wakeResult): the speculative activation has already started feeding STT when arbitration resolves. On a greeting win the wake loop cancels that speculative activation (discarding the wakeword tail + pre-greeting silence it captured) and starts a fresh activationCycle carrying the greeting clip. That activation's activationLoop plays the greeting as its first span (greeting.speak, before opening the listen window): playGreetingSpan calls Session.SendTTS (blocks until the satellite confirms playback drained), drains the audio conduit (the greeting's own AEC-cancelled near-silence buffered during playback), then STT opens on the command. The greeting thus shows in the Inspector as a real duration bar on the command turn, and the wake-time speaker is stamped as a speaker lane marker at t=0. Because the gate is server-side, satellites (ESP32/Android/desktop) need no changes.

Identity timing: the wake speaker comes from IdentifyWakeword on the carved lead. The capture open fires the WakeEvent before its trailing preroll chunks arrive, so the arbiter's LeadPCM (dispatcher waitLead) blocks until the full lead is carved — otherwise every wake identifies as "unknown" off an empty snapshot. The coalesce window is skipped when only one satellite is connected (no contention), so a single-satellite greeting starts right after identify instead of after the window. The end-of-STT speaker.identify span (turn-level identify_turn, for memory/routing) is a separate mechanism and still runs every turn.

Gotcha: do not feed the greeting window to STT — a half-second of greeting echo can trip a spurious TurnDetected. Playing the greeting before STT opens sidesteps it (no audio reaches the sidecar during the greeting); the conduit drain prevents the buffered playback window from polluting the first post-greeting frames. The greeting wav is never synthesised on the wake path — only the on-disk cache (${ALEPH_DATA_DIR}/greetings/<id>.wav, baked by the WebUI button / self-heal reconcile) is played; a missing or stale wav silently falls through to plain listening (logged, never silent).

Multi-room: external sink output routing (UPnP/DLNA, Cast)

Each satellite's output can be routed to a different sink instead of its own speaker. Sinks share one namespace: sat:<deviceID>, upnp:<UDN>, cast:<id>.

Discovery is subnet-only + trust-gated. SSDP/mDNS multicast does not cross VLAN/subnet boundaries, so the server only finds renderers on its own L2 segment. A discovered device is online but not routable until adopted in the WebUI — private TTS is never auto-routed to an unauthenticated LAN device. SinkIDs are stable across reboots (UPnP UDN, Cast id); settings map keys are a dot-free encoding of the SinkID because the settings path splitter treats . as a separator.

UPnP/DLNA sink gotchas (the ones that cost hours)

• L16 is big-endian. RFC 2586 audio/L16 is S16 big-endian — the opposite of our internal S16LE. The encoder byte-swaps each sample; skipping the swap is silent garbage / loud static. (WAV bodies stay little-endian.)
• Infinite WAV length. The live stream is open-ended, so the RIFF and data chunk sizes are written as 0xFFFFFFFF. The container header is sent eagerly on connect (before any PCM) — a renderer needs it to start, and a connection may sit in silence before the first utterance.
• DLNA live flags. contentFeatures.dlna.org uses DLNA.ORG_OP=00 (no seek) + the live DLNA.ORG_FLAGS, and the DIDL item is object.item.audioItem.audioBroadcast (non-seekable). transferMode.dlna.org: Streaming.
• LAN host, not loopback. The stream URL handed to a renderer must be a reachable LAN IP — never 127.0.0.1/0.0.0.0 (which webui/calendar/spotify coerce to loopback). sinkhttp.StreamHostFor uses the UDP-dial-to-renderer source-IP trick (correct on multi-homed/VLAN hosts); override with audio.routing.stream_advertise_host.
• Separate LAN server. The sink stream server binds a LAN interface (audio.routing.stream_listen_addr:stream_port, default 0.0.0.0:5604), NOT the loopback WebUI server.
• Fixed stream rate. Everything is normalised to 44.1 kHz so the persistent WAV header's rate never changes between a 24 kHz TTS and a 44.1 kHz media Play — the router resamples each source to the sink's cap; the HTTP pipeline only upmixes mono→stereo.
• GENA vs poll. Re-arm (SetAVTransportURI+Play) on STOPPED/NO_MEDIA_PRESENT uses GetTransportInfo polling as the primary path. KEEP_ALIVE injects short silence when idle so the renderer connection never underruns.
• Windows Firewall. The stream port needs an inbound allow rule on the Windows dev host, or renderers can't reach the stream.

Per-utterance latency

External renderers buffer before playback (~1–3 s on UPnP). The persistent-stream model pays that once at adopt time; each utterance then plays with only network + device jitter-buffer latency. A satellite's own speaker is always lowest-latency.

Media (Spotify) fan-out

When a satellite's output is assigned to a sink other than itself, its Spotify playback fans out through the same router as TTS. v1 limitations (the media path is otherwise stereo + per-device, the router/sink path mono):

• Mono on routed sinks. Routed music is downmixed to mono (then upmixed to stereo for UPnP renderers). The solo-satellite path is unchanged — direct, full stereo, no regression. Only external assignments lose stereo.
• Transport controls. Stop tears the fan-out down. Pause/volume are not yet plumbed per-member for routed media; treat routed external members as stop/restart-only (UPnP RenderingControl volume is a follow-up).
• TTS-vs-media suppression stays keyed to the source satellite, so TTS is still suppressed while routed music plays.

Android kiosk AEC + barge-in (Echo Show 5 / Lineage / MT8163)

AEC is client-side. The kiosk runs WebRTC AEC3 in-app (EchoCanceler, native libaleph_aec), fed the TTS PCM as the reference via TtsPlayer.pushReference. It was previously disabled for the echo-show-5 profile on the assumption the FireOS MTK HAL does hardware AEC — but on this Lineage build that HAL stage is a no-op (AudioALSACaptureHandlerAEC, besrecord_scene=-1), so the device had zero echo cancellation. It is now re-enabled (effectiveAec = echoCanceler); the wake detector runs on the cancelled mic.blocks (idle = pass-through, so cold wake sensitivity is unchanged), the meter + STT send on the same stream.

Voice barge-in is intentionally NOT implemented — barge-in is tap-only. Two approaches were tried and both fail on this hardware:

• Wakeword during TTS: AEC3's nonlinear residual suppressor distorts the near-end voice during double-talk; the classifier never crosses 0.3 even though the cancelled stream audibly carries the user.
• Near-end energy: AEC3 cancellation is too inconsistent (ERLE swings ~8–45 dB block-to-block) because the reference we feed has timing jitter. Loud-TTS blocks where AEC briefly fails leak residual echo at the same level as user speech, so an energy detector self-aborts the assistant's own response.

Root inspection (adb root) confirmed the real blocker: the app already sends the MTK enable params (HDREC_SET_VOICE_MODE=1;SET_AECREC_TEST_ENABLE=1, seen in dumpsys media.audio_flinger) but the stock vendor HAL ignores them (no BesRecord scene loads). The hardware AEC infra is all present (libspeech_enh_lib.so, Tap_AEC_mic{1,2}.cfg, Audio_ExtCodec_EchoRef_Switch

• codec loopback in tinymix), and libaudiopreprocessing.so exists, but audio_policy_configuration.xml declares no echo-reference input device, so the AOSP software-AEC route has no reference either. Reliable HW AEC would need a vendor audio-HAL patch (act on the enable params / load BesRecord), which is a build-time vendor-binary change, not a runtime tweak. A possible future path is a root helper capturing the DL1_AWB_Record loopback PCM as a hardware-aligned reference for AEC3 — untried, large.