Designing and building a voice assistant that lives on my Mac — on-device speech, a real memory, and hands on the machine

Conversation turns and durable facts are stored in SQLite with WAL journaling; memory writes survive crashes.

After each exchange, a background task extracts new facts about me — never delaying the spoken reply.

Everything she knows is loaded into her system prompt when she starts.

Both are first-class tools she chooses to use herself — not keyword triggers.

Opens apps by name or alias (“open my browser” → Arc), opens URLs and deep links, reads system vitals, and controls volume and media.

Her own Playwright browser window with a persistent profile, so logins survive. Navigate, read, click, and fill by voice — anything that submits is hard-gated behind my verbal confirmation.

“Look at my screen” captures, downscales, and feeds her vision — she answers the question asked instead of inventorying the pixels.

She detects the frontmost app, the active Arc tab's URL, or the open document, and summarizes it on request.

PDFs, Word documents, Excel sheets, and plain text; views images.

PDF, Word, Excel, and text files on the Desktop from a voice request — structured headings and tables, filenames derived from the content, and existing files never overwritten.

An upload button in the HUD hands her any file; she acknowledges it and waits for the next instruction before acting on it.

She drafts messages to my saved contacts with the text pre-filled. She cannot press send — only I can.

The rule lives in the tool layer, not in a prompt suggestion. The same pattern gates destructive browser actions behind spoken confirmation.

Live weather from Open-Meteo for Riyadh or any city, feeding both her voice and a HUD card.

Live web search through Brave's API.

“Set a timer for ten minutes” — she announces when it's up, waiting for a quiet moment instead of cutting in.

“What did I just say?” gets a verbatim replay, and boot opens with a time-aware greeting.

Slow rotation at rest, rings pulsing to my mic level while listening, a fast sweep while thinking, and a core that throbs to the actual speech amplitude while she talks — driven by live audio data over WebSocket, not canned animation.

Telemetry, weather, session stats, and an optional camera card on the left; the orb in the center; the conversation panel with typed input and file upload on the right; a collapsible mission log along the bottom, showing every state change and tool call as it happens.

Every color, glow, radius, font, column width — even the layout's max-width — lives in a single tokens file. The interface's entire personality reskins by editing CSS variables; the previous theme, near-black with cyan glows, is preserved in a comment.

If the backend link drops, the whole theme flips to a signal-red state.

A typed-line startup animation with a two-tone chime synthesized in WebAudio — no audio assets — then her spoken greeting.

A global hotkey (⌘⇧J), a full typed input path beside voice, and the camera off by default.

The TTS SDK's audio helpers needed system dependencies that didn't exist on the machine — solved by streaming raw PCM straight to the audio device.

She kept hearing herself through the speakers. Solved with runtime calibration against her own speaker bleed.

The first weather API geocoded Riyadh to South Africa. Replaced.

No package manager, an ancient system Python, thread-unsafe UI frameworks, zombie processes — handled one at a time.

Speech recognition is 100% local; audio never leaves the machine.

The AI backend authenticates through my existing Claude sign-in — no API key exists in the project.

Secrets live in .env and are deny-listed from her file access — she refuses to read them even when asked.

The wake-word mic sampler runs only when explicitly armed, and only while she is otherwise idle.