Homelab Voice Assistant (V.A.U.L.T.)

A voice-first console for one machine, one home, and one homelab. You speak; the right thing happens in about a second; the screen shows what was heard, which tier decided, which tool ran, and what changed. Everything on screen is live or labeled as not live.

The stack is local and subscription-only: whisper.cpp or Kyutai for speech to text, a tool-calling model on Ollama for the brain, Kokoro for speech, and the Claude Code CLI (claude -p) on the owner's subscription for heavy work. There is no Anthropic API key.

Setting it up on your machine

This repo describes no household. Entity ids, LAN addresses, the service list and the port map all live in one file outside the repo — ~/.agentic-os-hud/profile.json — so a fresh clone starts empty and connects to nothing until you tell it what you have.

The short way. Open this folder in Claude Code and say:

set up the HUD

It runs .claude/skills/setup-hud: checks what you already have installed, asks for your Home Assistant address and a long-lived token, lists your real entities and asks which one is the plug, which is the TV, and what you call them out loud, finds your containers and open ports and asks which belong on the lab panel, then writes and validates the profile and drives the result.

The manual way. Copy config/profile.example.json to ~/.agentic-os-hud/profile.json and edit it — every field is documented inline and every one is optional. Then:

node scripts/setup/setup.mjs doctor        # what is installed, running and configured
node scripts/setup/setup.mjs ha-entities   # your entity ids, grouped by domain
node scripts/setup/setup.mjs show-profile  # read back what is configured

Secrets never go in the profile or in .env.local: tokens live in their own 0600 files under ~/.agentic-os-hud/ and are written by setup.mjs save-token.

What you need. Node 22+ and uv for the voice services; Home Assistant and Ollama are optional but most of the point. Without a profile the HUD still builds, boots, and passes its tests — it just has nothing of yours to show.

How a turn is handled

A turn is one utterance (or one typed line) and its reply. The voice server routes each turn through three tiers, cheapest first:

Tier	What	Where	Reaches first audio in
T0 intents	An ordered table of deterministic patterns: stop and cancel, time and date, turn on/off/toggle <alias>, mute/unmute/volume up/down the tv, open/quit <app>, set volume to N, brief, search the web for X, run <skill>, status. No model.	voice/intents.py	about 0.7 s (budget from measured parts)
T1 brain	qwen35-tools over Ollama /api/chat with native tools, think: false, up to 3 tool rounds. Tools: home_control, home_status, mac_control, mac_search, system_status, lab_status, lab_control, web_search, vault_search, run_skill, delegate_to_claude, remember. Every tool returns {ok, say, data}; say is the receipt the HUD prints and what Jarvis speaks if the model says nothing.	voice/brain.py, voice/tools/	1.15 to 1.45 s (budget from measured parts)
T3 lean Claude	claude -p started without hooks or MCP servers (VOICE_CLAUDE_LEAN=1), permission mode from VOICE_T3_PERMISSION, reached through the delegate_to_claude tool, the console's claude chip, or a skill button.	voice/server.py, app/api/run-skill	2.1 s to start, then the task

The existing named-agent patterns and media routing (voice/route.py) run ahead of the brain and are unchanged. The vault answer path (qmd over the Obsidian vault) is a T1 tool.

Measured on the owner's Mac mini (M2 Pro, 16 GB) on 2026-09-05:

Part	Measured
Browser VAD end-of-speech tail	about 768 ms with library defaults, tuned to about 300 ms
whisper.cpp base.en	0.45 s for a 3.5 s clip
Kyutai streaming STT	keeps pace live; finalize 0.6 s with a 2 s pad (now 0.6 s); 0.8 GB RSS
qwen35-tools	0.36 s to first token warm; correct Home Assistant tool call in 0.6 to 1.6 s
Kokoro	0.26 to 0.34 s per short sentence after warm-up
claude -p	80 s wall with the full profile (hooks plus 30 MCP servers); 2.1 s lean

The HUD measures every turn (metrics frame: stt_ms, first_token_ms, first_audio_ms, total_ms) and shows the last-turn latency in the top bar, so the budget above is checked live.

Ports

Port	Service	Bound by
3001	Next.js web app (/api/*, the page)	HUD_WEB_HOST:HUD_WEB_PORT, default 0.0.0.0:3001
8765	Voice WebSocket (voice/server.py)	VOICE_HOST:VOICE_PORT, loopback unless VOICE_ALLOW_LAN=1
8766	Kokoro TTS (voice/tts_server.py)	TTS_HOST:TTS_PORT, loopback unless VOICE_ALLOW_LAN=1
8123	Home Assistant (Docker)	HUD_HA_URL
11434	Ollama	HUD_OLLAMA_HOST:HUD_OLLAMA_PORT
8888	SearXNG (Docker)	HUD_SEARXNG_URL

Environment variables

.env.example documents every variable with its default. The ones added for v2:

Variable	Default	Read by
HUD_WEB_HOST / HUD_WEB_PORT	0.0.0.0 / 3001	scripts, launchd, lib/config.ts, the run_skill tool
HUD_HA_URL	http://127.0.0.1:8123	lib/ha.ts, voice/tools/ha.py
HUD_HA_TOKEN_FILE	$HOME/.agentic-os-hud/.ha-token	same (read at runtime, never logged)
HUD_SEARXNG_URL	http://127.0.0.1:8888	voice/tools/search.py, /api/health
HUD_DECK_SERVICES_PATH	$HOME/.agentic-os-hud/services.jsonc	lib/lab.ts (optional; the profile supplies services)
HUD_OLLAMA_MODEL_BRAIN	qwen35-tools:latest	voice/brain.py
HUD_OLLAMA_KEEP_ALIVE	-1	voice/brain.py, warm-up
VOICE_T3_PERMISSION	bypassPermissions	voice/server.py
VOICE_CLAUDE_LEAN	1	voice/server.py
STT_BACKEND	whisper everywhere; kyutai is opt-in (streams, 0.2 s finalize, 2 GB resident)	voice/stt.py
KYUTAI_TRAIL_PAD_S	0.6	voice/stt.py
VOICE_ALLOW_LAN	unset (loopback); 1 in the launchd plists	voice/config.py

Running it

Prerequisites: Node 20.9 or newer (this machine runs 24), Python 3.12 with uv, Ollama with the brain model in ollama list, whisper.cpp (brew install whisper-cpp) or the Kyutai group (cd voice && uv sync --group stt-kyutai --group dev), the Kokoro model files, and the Claude Code CLI logged in. Home Assistant needs a long-lived token at ~/.agentic-os-hud/.ha-token (mode 0600).

Always on, as three launchd user agents (com.agenticos.hud, com.agenticos.voice, com.agenticos.tts):

bash ops/launchd/install.sh      # renders ops/launchd/*.plist with this repo's path, bootout + bootstrap
launchctl list | grep agenticos  # a number in the first column is a live PID; "-" with an exit code is a crash loop
bash ops/launchd/uninstall.sh

The plists set PATH, HOME, and the v2 variables above; the web agent runs ops/launchd/run-web.sh (builds once when .next/BUILD_ID is missing, then npm run start -- -H 0.0.0.0 -p 3001), the voice agent runs voice/run.sh, the TTS agent runs voice/tts-run.sh. Logs: ~/Library/Logs/agentic-os-hud/{web,voice,tts}.log. ops/setup/update.sh pulls, rebuilds, and reinstalls the agents in the right order.

Dev loop (foreground services, each skipped when already listening):

npm install
cp .env.example .env.local        # then edit
bash scripts/start-hud.sh         # ollama, tts, voice, next dev on 0.0.0.0:3001
bash scripts/stop-hud.sh

Open http://127.0.0.1:3001. Press the mic once and talk (it keeps listening; talking over a reply stops it), hold Space for push-to-talk, or type in the ledger's command line; Esc clears it.

Testing it

npx tsc --noEmit                      # types
npm run lint                          # eslint + tsc
npm test -- --run                     # vitest
npm run build                         # production build
cd voice && uv run pytest -q          # voice pipeline, intents, brain, tools, barge-in
bash scripts/check-docs-drift.sh      # docs vs code (header names, token, permission flags)
bash ops/setup/smoke-test.sh          # live: /api/health, TTS round trip, pytest, Ollama tags

Live checks once the stack is up:

curl -s 127.0.0.1:3001/api/health | jq '.integrations'
curl -s 127.0.0.1:3001/api/home | jq '.source, (.entities | length)'     # "live", > 50
curl -s 127.0.0.1:3001/api/system | jq '.load, .mem'
# a text turn over the voice WebSocket, no microphone needed:
voice/.venv/bin/python - <<'PY'
import asyncio, json, pathlib, websockets
async def main():
    token = pathlib.Path.home().joinpath(".agentic-os-hud/.session-token").read_text().strip()
    async with websockets.connect(f"ws://127.0.0.1:8765/?token={token}") as ws:
        await ws.send(json.dumps({"type": "text", "text": "what time is it"}))
        while True:
            m = await ws.recv()
            if isinstance(m, bytes): continue
            f = json.loads(m); print(f)
            if f.get("state") == "metrics": break
asyncio.run(main())
PY

HTTP routes

All under app/api/. Read routes check the request origin; mutating routes also require the per-process x-hud-token header (minted in lib/session-token.ts, stored at ~/.agentic-os-hud/.session-token, injected into the page).

Route	Purpose
GET /api/home, GET /api/home/stream (SSE), POST /api/home/call	Home Assistant entities and summary; live state_changed fan-out; allowlisted service calls (switch, media_player, button)
GET /api/lab, POST /api/lab/control	hosts, Docker containers, service probes; restart a container or fetch its last 60 log lines
GET /api/system	load, CPU, memory, disk, network, power, top processes
GET /api/health	Ollama, voice, TTS, Claude CLI, Home Assistant, SearXNG reachability
POST/GET/DELETE /api/run-skill, GET /api/run-skill/stream	claude -p jobs with SSE output
/api/agents, /api/routines, /api/schedule, /api/tokens, /api/projects, /api/morning-brief, /api/youtube, /api/session-token	existing job, routine, usage, project and briefing routes

/api/home answers source: "offline" with an empty entity list when Home Assistant is unreachable; nothing is invented.

Voice WebSocket protocol

The upgrade needs ?token=<HUD session token> (the value the web app writes to ~/.agentic-os-hud/.session-token at start and bakes into its page); the server answers 403 without it, and refuses browser origins outside loopback, private LAN ranges and Tailscale. A plain HTTP GET on the port still returns the liveness JSON. The page adds the token itself.

Client to server: binary PCM16 16 kHz frames, then {"type":"eou"}; or {"type":"text","text":"...","model":"auto","mode":"plan","effort":"medium"} for a typed turn; reset; barge_in. Server to client: listening, thinking, speaking (with the growing reply), visemes, plus per-turn heard (transcript and stt_ms), routed (tier and label), tool (name, args, running|done|error, result, ms), and metrics. Turn ids are per-connection (t_0001, ...). Audio comes back as PCM 24 kHz. A newer connection supersedes an older one; a typed command while speaking acts as barge-in first.

Security

• The voice WebSocket requires the per-start session token in its URL and an allowed browser origin (voice/server.py, authorize_request); without the token file it refuses every upgrade. The socket can drive Home Assistant, Docker, this Mac and Claude, so it is gated like a mutating route.
• Every /api/* route checks the Origin; mutating routes also require the x-hud-token header and an allowed Origin (loopback, private ranges, Tailscale CGNAT and MagicDNS; lib/run-skill-helpers.ts). Cross-origin requests get 403.
• The voice and TTS services bind loopback unless VOICE_ALLOW_LAN=1. The web app binds HUD_WEB_HOST (default 0.0.0.0 so a phone on the LAN can load it); set 127.0.0.1 to keep it local.
• Free-text Claude runs default to plan (read only); acceptEdits and bypassPermissions need the per-request confirmEdits flag from the confirm dialog. The voice server's delegated runs use VOICE_T3_PERMISSION, default bypassPermissions (owner decision for this single-user console; change the variable to lower it).
• Tokens live in 0600 files under ~/.agentic-os-hud/ (.ha-token, .session-token, .claude-token) and are never logged or committed.
• Mac control runs osascript, open, mdfind, pmset under a 4 s timeout; a permission prompt becomes an error sentence, not a hung loop. Lab control targets only names from the current docker ps.

Layout

• app/page.tsx: the sonar room (top bar, ledger, scope, rail). app/api/: routes above.
• components/: Scope, Ledger, PhosphorTrace, TopBar, VoicePanel (mic, VAD, WS client, playback), rail/ (Home, Lab, System, Skills), plus the existing routines, agents, project cards, boot briefing, run cards and dialogs.
• hooks/: useLedger, useHomeState (SSE), useJobStream, useDispatch.
• lib/: config.ts (every process.env read), ha.ts, lab.ts, system.ts, contacts.ts, voiceState.ts, run-skill-helpers.ts, session-token.ts.
• voice/: server.py (WS, Ollama and Claude streams, warm-up), ws_handlers.py (dispatch, barge-in), intents.py, brain.py, tools/, stt.py, tts_client.py, tts_server.py, config.py, tests test_*.py.
• ops/launchd/: plist templates and scripts. ops/setup/: client setup, update, smoke test.
• scripts/: dev start and stop, docs drift check, ha-set-timezone.py.

Troubleshooting

• launchctl list | grep agenticos shows - and an exit code: read ~/Library/Logs/agentic-os-hud/<service>.log. Exit 126 or "No such file" means the plist points at a path that does not exist; rerun bash ops/launchd/install.sh from the repo.
• /api/health says Home Assistant is unreachable: check HUD_HA_URL, that the token file exists with mode 0600, and curl -H "Authorization: Bearer $(cat ~/.agentic-os-hud/.ha-token)" 127.0.0.1:8123/api/ answers "API running.".
• Brain turns fail: ollama list must contain HUD_OLLAMA_MODEL_BRAIN; ollama ps shows whether it is resident.
• T3 reports "not logged in" under launchd: run claude setup-token and store the token at ~/.agentic-os-hud/.claude-token.
• Memory pressure (vm_stat compressor growing, sysctl vm.swapusage climbing): the brain's first token stretches from 0.4 s to over 1 s while model pages re-fault. Kyutai STT is the usual culprit (2 GB physical); keep STT_BACKEND=whisper unless the box has headroom.

Dependency note

onnxruntime-web has no static import in app/, components/, or lib/; @ricky0123/vad-web loads it at runtime from public/vad/. Keep it in package.json.

Rollback

git checkout pre-jarvis-v2 (tag on 28796fb), then bash ops/launchd/install.sh from that tree. The Home Assistant time zone change and the archive move under ~/Desktop/Projects/Homelab Voice Assistant/ are independent and reversible (scripts/ha-set-timezone.py Atlantic/Reykjavik; move the archive back and remove the symlink).