Skip to content
PrivateAI
← Back to Home
Guides

Build a Private AI Voice Assistant: The No-Cloud Alexa Alternative

10 min read min readBy PrivateAI Team

_Last updated: 2026-07-04_

Here is the short version: you can replace Alexa or Google Assistant with a voice assistant that never sends a single audio clip off your network. Wake word detection, speech-to-text, the reasoning that decides what you meant, and the voice that answers you all run on hardware you own — a Home Assistant server, a local Whisper model, Ollama, and a Piper text-to-speech engine, talking to small microphone satellites scattered around your house.

No recordings sit on a corporate server waiting for a contractor to review them. No "smart speaker" subpoena can produce a transcript of your kitchen at 7am. If your internet goes down, your lights still turn on when you ask.

This guide walks through the full stack, the hardware you need, and where a fully local setup still has a real gap — and what to route to a cloud service instead of pretending the gap doesn't exist.

Why Cloud Voice Assistants Are a Privacy Problem You Already Know About

This part doesn't need convincing so much as a reminder of the specifics. Amazon has confirmed, in testimony to Congress, that Alexa recordings are reviewed by human annotators to improve transcription accuracy. Google has had the same practice with Google Assistant. Both companies have said users can opt out of review — but the recordings are still made, still uploaded, and still retained on a schedule you don't control.

The deeper issue isn't the review program. It's the architecture. A cloud voice assistant's entire design assumes your audio needs to leave your house to be useful. Every "Alexa, what's the weather" trip travels to a data center, gets transcribed, gets logged with a timestamp and device ID, and becomes a data point in a profile that also includes your shopping habits, your calendar, and — if you've linked accounts — your location history.

None of that is required to turn on a light or set a timer. It's required because the business model is data collection with a voice interface bolted on, not the other way around.

The Local Architecture, In Plain Terms

A privacy-respecting voice assistant needs four things, and every one of them can run on hardware sitting in your closet:

  1. Wake word detection — listens for "Hey Jarvis" (or a custom name) and does nothing else until it hears it. Runs on the microphone satellite itself, using a tiny model like openWakeWord — a few megabytes, no network needed.
  2. Speech-to-text (STT) — converts your spoken command to text once the wake word fires. Runs locally via faster-whisper, a fast implementation of OpenAI's Whisper model, hosted on your home server rather than OpenAI's.
  3. Intent handling / reasoning — decides what you meant and what to do about it. Simple commands ("turn off the lights") are handled by Home Assistant's built-in intent engine. Open-ended questions get routed to a local LLM through Ollama.
  4. Text-to-speech (TTS) — speaks the answer back, using Piper, a fast local TTS engine with natural-sounding voices that runs on a Raspberry Pi without breaking a sweat.

All four pieces are tied together by Home Assistant's Assist pipeline, which is the free, open-source orchestration layer that replaces the "Alexa" or "Google" part of the stack. Your microphone satellites are the only devices that need to be near you — everything else runs on a central box.

Hardware You Actually Need

You don't need a GPU farm for this. A voice assistant pipeline is lightweight compared to, say, coding-assistant workloads.

Central server (runs Home Assistant, Whisper, Ollama, Piper):

  • A mini PC or NAS with 16GB+ RAM is comfortable. An old laptop works for testing.
  • If you're also running a larger LLM for other tasks, see our private AI home server guide for hardware sizing.

Microphone satellites (one per room where you want voice control):

  • Home Assistant Voice PE — the official $59 satellite device, purpose-built for this exact pipeline, with a physical mute switch and no cloud dependency.
  • M5Stack Atom Echo (~$13) — a cheap DIY option, flashed with ESPHome's voice assistant firmware.
  • Seeed ReSpeaker boards — better microphone arrays for larger rooms, at a higher price point.

The point of a satellite architecture: the always-listening component is a $13–$60 device with a physical mute switch, not a $400 phone with an always-on microphone and a data connection to three different companies.

Step 1: Install Home Assistant and the Voice Add-ons

Install Home Assistant OS on your central server (a Raspberry Pi 5, mini PC, or a VM works). Once it's running:

  1. Go to Settings → Add-ons → Add-on Store
  2. Install Whisper (the local STT add-on, built on faster-whisper)
  3. Install Piper (the local TTS add-on)
  4. Both add-ons run in Docker containers managed by Home Assistant — no manual model downloads required

Under Settings → Voice Assistants, create a new Assist pipeline and select your Whisper and Piper add-ons as the STT and TTS engines. Test it from the Home Assistant mobile app's microphone icon before moving to physical satellites.

Step 2: Add Ollama as Your Conversation Agent

Home Assistant's default intent engine handles device control well ("turn on the office lights") but falls flat on anything conversational. This is where a local LLM comes in.

Install Ollama on your server (or a separate machine on your network — see our Tailscale guide if you want to run it elsewhere and reach it remotely) and pull a small, fast model suited to quick responses:

```bash

ollama pull llama3.2:3b

```

A 3B model responds in under a second on modest hardware, which matters for voice — nobody wants to wait four seconds for their thermostat to argue with them.

In Home Assistant, install the Ollama integration (Settings → Devices & Services → Add Integration → Ollama), point it at your server's address, and select your pulled model. Then, in your Assist pipeline, set the conversation agent to "Ollama" instead of the default.

Now commands like "is it going to rain later" or "remind me why I turned off the water heater" get a real, reasoned answer instead of "Sorry, I didn't understand that."

Step 3: Flash and Place Your Satellites

If you're using the Home Assistant Voice PE, it auto-discovers on your network — plug it in, and Home Assistant prompts you to add it.

For a DIY ESP32-based satellite (like the Atom Echo), flash it using ESPHome's web installer with the pre-built "voice assistant" configuration. This takes about five minutes and requires no code changes — just a USB cable and a browser.

Place satellites where you'd naturally talk: kitchen, bedroom, home office. Each one connects back to your Home Assistant server over your local Wi-Fi. Audio never touches an external network unless you've explicitly configured a fallback (more on that below).

The Honest Gap: What Local Can't Do Well

A 3B–8B local model is genuinely good at intent parsing and short factual answers. It is not going to compete with a frontier model on open research questions, and it has no access to live information — no current weather (that comes from your existing Home Assistant weather integration, not the LLM), no live sports scores, no "what happened in the news today."

Pretending a local voice assistant can do everything leads to the same outcome as most all-local setups: people quietly reach for their phone and ask Siri or Google anyway, undoing the entire privacy benefit out of habit.

The better answer is to be deliberate about the one narrow slice of queries that genuinely need external, current information, and route only those to a tool that respects your privacy more than Alexa does — rather than funneling everything through a system built around ambient data collection.

Perplexity is a reasonable choice for that narrow slice. It's built as a search-and-cite tool rather than an ambient assistant: it doesn't require linking a home profile, doesn't build a voice-print history, and its paid tier doesn't train on your queries. For "what's today's top headline" or "who won last night's game" — the kind of thing your local model can't know — a quick Perplexity lookup triggered through a Home Assistant script (using its API) is a narrower privacy exposure than routing your entire household's conversation through Alexa's always-on microphone.

Affiliate Disclosure: This article may contain affiliate links. If you make a purchase through these links, we may earn a small commission at no extra cost to you. We only recommend products we genuinely believe in. This helps support our work and allows us to continue providing free content.

The workflow is simple: Home Assistant's built-in backup feature exports a .tar archive of your full configuration on a schedule you set. Point that backup folder at a synced Tresorit folder instead of Dropbox or Google Drive, and your smart-home credentials stay encrypted end-to-end, even in the cloud copy.

Testing the Full Loop

Once everything is wired up, test each layer separately before trusting the whole chain:

  • Wake word: say your wake word from across the room — it should trigger a chime or light change on the satellite within a second
  • STT accuracy: check the transcribed text in Home Assistant's Assist debug log matches what you said
  • Intent routing: a device command ("turn off the lamp") should execute instantly; an open question should visibly hand off to Ollama in the log
  • TTS response time: Piper should respond within 1–2 seconds on a 3B model; if it's slower, your server may be CPU-bound and would benefit from a smaller model or better hardware

Common Pitfalls

Wake word triggers randomly. openWakeWord's default sensitivity can be too aggressive in noisy rooms. Lower the sensitivity threshold in the satellite's ESPHome configuration.

Ollama responses feel sluggish for voice. Larger models (8B+) trade speed for capability. For voice, prioritize a smaller, faster model — the conversational bar for a voice assistant is lower than for a chat interface, since responses are shorter and users are more forgiving of simple answers delivered instantly.

Satellite loses Wi-Fi connection intermittently. ESP32-based satellites are sensitive to 2.4GHz interference. Keep them on a dedicated IoT network segment if your router supports it — this also limits what a compromised satellite could reach on your network.

"Set a timer" works but "what's my day look like" doesn't. This is expected — calendar integration requires connecting Home Assistant to your calendar source separately from the voice pipeline. It's not a limitation of the LLM, it's a missing integration.

The Setup in Three Sentences

Install Home Assistant with the Whisper and Piper add-ons for local speech-to-text and text-to-speech. Add Ollama as your conversation agent for anything beyond simple device commands. Place ESP32 or Home Assistant Voice PE satellites around your house, and route only the narrow slice of queries your local model can't answer — current events, live scores — to a privacy-respecting tool like Perplexity instead of a full cloud assistant.

The whole build takes a weekend. The result is a voice assistant that answers to you, and only you.

Keep Learning

For sizing the server that runs this stack alongside other local AI workloads, see our private AI home server guide. If you want to reach your Home Assistant and Ollama setup securely while away from home, our Ollama + Tailscale guide covers remote access without opening ports. For hardening the server itself, see our Ollama network security guide.

Want a weekly digest of private AI tools, setup guides, and privacy news — without the newsletter tracking pixels?

Join the PrivateAI list. Plain text. Unsubscribe any time.

Stay Updated

Join our newsletter for the latest updates.