Wake-Word Voice Assistant.

Project Overview

Listens for its wake word entirely on-device and handles voice commands offline — always listening, never sending your audio to the cloud.

Voice assistants feel magical but carry a real privacy cost: many are always listening and stream audio to the cloud to understand it. This project builds a voice assistant that keeps the listening — and ideally the understanding — on the device: it detects a wake word ("Hey Jarvis") entirely on-device, and handles a set of voice commands offline, so your audio never has to leave the room. It is a compelling embedded-AI project because it must run capable speech models within the tiny compute and power budget of a microcontroller or single-board computer.

The heart of it is wake-word detection, a beautifully constrained problem. A tiny, efficient neural network runs continuously on the incoming audio, listening for one specific phrase — and it must be extremely cheap (so it can run always-on within a small power budget) yet accurate in two directions: few false rejections (it should wake when you say the word) and, critically, few false accepts (it must not wake at random sounds). Only after the wake word fires does the (heavier) command-recognition stage run, so the expensive work happens rarely. Keeping both stages offline — wake word plus a set of local commands — is what preserves privacy and gives instant, network-independent response.

The value is a private, responsive, offline voice interface — for smart-home control, appliances, or accessibility — that isn't always phoning home. It is honest about the hard engineering: fitting speech models into severe compute/memory/power limits (this is TinyML), the accuracy trade-offs of a tiny model, robustness to noise and accents, and the reality that a small offline vocabulary is limited compared to a full cloud assistant. It is also candid that "always listening" deserves genuine privacy care even when local. Built well — an efficient always-on wake-word model, a two-stage design, and offline command handling — it is both a genuinely private voice assistant and the definitive lesson in on-device, resource-constrained speech AI.

What this project does

• Detects a wake word continuously on-device
• Handles a set of voice commands offline
• Keeps audio local — no cloud round-trip
• Runs a tiny always-on model within a small power budget
• Uses two stages: cheap wake word, then command recognition
• Responds instantly, network-independent
• Preserves privacy by design

Real-World Applications

Deployment contexts where a build of this kind earns its keep.

Features & Capabilities

• On-device wake-word detection (always-on)
• Offline command recognition
• Two-stage (cheap detect → command) design
• Low false-accept / false-reject tuning
• TinyML resource-constrained models
• Privacy-preserving (audio stays local)
• Honest about small-vocabulary and noise/accent limits

Difficulty, Time & Required Skills

Skills you should have (or will pick up)

• Wake-word detection (efficient always-on models)
• On-device / offline speech command recognition
• TinyML (fitting models to MCU/edge limits)
• False-accept/false-reject tuning
• Privacy-preserving audio design

Bill of Materials

Every part below is commonly available from Indian and international hobby-electronics suppliers. Prices are indicative 2026 retail figures in Indian rupees and will drift — treat them as a budgeting guide, not a quotation.

Estimated total: ₹1,520, excluding tools, shipping and consumables.

Tools and consumables

• Soldering iron (temperature controlled, 350 °C) with 0.8 mm 60/40 or lead-free solder
• Digital multimeter — continuity, DC volts and current ranges
• Wire strippers, flush cutters and a small set of precision screwdrivers
• Heat-shrink tubing and a heat gun (or a lighter, carefully)
• A laptop with a USB port and the toolchain listed above

Hardware Specifications

Consolidated electrical and interface specifications for every active part in the build.

Power Budget & Supply Sizing

Add up the typical active current of every part, then size the supply with at least 50 % headroom so transmit bursts and motor inrush never brown out the controller.

Summed typical draw is 181.4 mA. With a 1.5× design margin the supply should deliver at least 300 mA continuously at the stated rail voltage.

Software Requirements & Development Environment

Reference toolchain: Python / C++ + TinyML / on-device speech. Anything newer normally works; anything older may lack the board definitions used here.

• Install the Arduino IDE 2.3.x (or PlatformIO if you prefer a real editor and dependency locking).
• Add https://espressif.github.io/arduino-esp32/package_esp32_index.json under File → Preferences → Additional Board Manager URLs, then install esp32 from the Boards Manager.
• Set the correct port under Tools → Port. On Linux add yourself to the dialout group: sudo usermod -aG dialout $USER and log out and back in.
• Open the Serial Monitor at 115200 baud — every sketch here logs its state there.
• Keep File → Preferences → Show verbose output during: compilation switched on while you are debugging build errors.

Required libraries

Block Diagram

The block diagram shows the functional decomposition of the system — what senses, what decides, what acts, and where the data ends up.

Circuit Diagram & Wiring

The "wiring" combines real audio hardware and a two-stage on-device pipeline — a microphone feeds an always-on wake-word model; only after it fires does offline command recognition run, all locally.

Wire one row at a time and tick it off — most "it does not work" reports trace back to a single swapped pair.

Wiring explanation

• A digital microphone feeds audio to the device (I2S/MEMS).
• A tiny wake-word model runs continuously (always-on) on the audio.
• Only after the wake word fires does the heavier command recogniser run.
• Everything stays on-device — no audio leaves for the cloud.
• Drive an output (LED/relay/speaker) for the response.

System Architecture

Read the stack from the bottom up: physical hardware, the firmware that drives it, the transport that moves data off the device, and the software a human actually looks at.

Working Principle

The defining goal of this project is privacy through locality: a voice assistant that does not stream your audio to the cloud. Many commercial assistants are always listening and send audio off-device to understand it, which is a genuine privacy exposure. Keeping both the listening (wake word) and the understanding (commands) on the device means the audio never has to leave the room — which also gives instant, network-independent response. Achieving this on tiny hardware is the whole engineering challenge, and it forces a smart architecture.

That architecture is a two-stage cascade, and it exists because of a fundamental tension: you want the device to be always listening, but you cannot afford to run a heavy speech model continuously on a microcontroller's power budget. The resolution is to split the work. Stage one is wake-word detection: a tiny, extremely cheap neural network runs continuously, doing nothing but listening for one specific phrase. Only when it fires does stage two — the heavier command recogniser — run, briefly, to understand what you actually said. Because the expensive stage runs only after a wake word (rarely), the device can be always-on within a small power budget. The wake word is the gate that makes always-on feasible.

The wake-word model has a sharply defined and demanding spec. It must be cheap enough to run always-on (few operations, tiny memory — TinyML territory), and accurate in two directions at once. A false rejection (failing to wake when you say the word) is annoying; a false accept (waking at a random sound, a TV, an unrelated phrase) is worse — it makes the assistant fire spuriously and, in a cloud assistant, would send audio unexpectedly. So the model is tuned to a careful operating point balancing false-accept and false-reject rates, and this trade-off is the central quality metric of wake-word systems. Getting a small model to reliably spot one phrase amid noise, without triggering on everything else, is the core difficulty.

The honesty this project requires is about the constraints and their consequences. Fitting speech models into a microcontroller's severe compute, memory and power limits is genuinely hard — this is embedded/TinyML engineering, where every kilobyte and milliwatt counts — and a tiny model is inevitably less accurate than a big cloud one, more sensitive to noise and accents, and limited to a small offline vocabulary of commands rather than open-ended conversation. That trade — a modest, local, private assistant versus a powerful, cloud-dependent one — is the honest bargain, and for many uses (a handful of smart-home commands, an appliance, an accessibility control) the local version is exactly right. It is also worth stating plainly that "always listening" warrants real privacy care even when local: the design keeps audio on-device precisely so that being always-on is not a surveillance risk. Built with an efficient always-on wake-word model, the two-stage cascade, and offline command handling, the assistant delivers genuine private, responsive voice control while teaching the substance of resource-constrained, on-device speech AI.

The maths behind it

Two-stage cascade (why always-on is feasible)

Stage 1 (always-on, CHEAP): wake-word model on every audio frame
Stage 2 (rare, heavier): command recogniser — ONLY after wake

cost ≈ always·cheap + rarely·heavy  → fits a small power budget.

Wake-word operating point

Tune the detection threshold to balance:
  false reject  (missed wake word) — annoying
  false accept  (wakes at random sound) — worse

Central metric: low FA/hour AND low FRR. Small model,
always-on, robust to noise.

Privacy by locality

wake word + commands run ON-DEVICE (offline)
  → audio never leaves the room
  → instant, network-independent response

TinyML limits → small vocabulary, lower accuracy (the bargain).

Program Flowchart

The firmware is a single cooperative loop. Nothing blocks for long, so networking, sensing and the user interface all stay responsive.

Assembly Instructions

Build on a breadboard first and only commit to solder once the whole system has run for an hour without a fault.

1. Build the audio front-end and wake-word stage

   Wire a digital microphone, extract features on-device, and run a tiny always-on wake-word model continuously.

2. Add offline command recognition (stage two)

   Run the heavier command recogniser only after the wake word fires, keeping it offline with a small vocabulary.

3. Tune and fit within TinyML limits

   Tune the false-accept/false-reject operating point, quantise/fit models to the device, and drive local outputs.

Step-by-Step Implementation Guide

Work through these in order. Each step ends in something you can observe, so a failure is always localised to the step you just finished.

1. Run the two-stage on-device pipeline

   Continuously run the cheap wake-word model; only after it fires, run offline command recognition and act locally.

   cppassistant.ino Copy Download

   const float THRESHOLD = 0.85f;         // low false-accept operating point
   
   void loop(){
     readAudioFrame(buf);                   // continuous, on-device
     computeFeatures(buf, feats);           // MFCC/log-mel
   
     float score = wakeWordModel(feats);    // STAGE 1: tiny, ALWAYS-ON
     if (score < THRESHOLD) return;         // keep listening cheaply
   
     // Wake word detected -> STAGE 2 runs only now (rarely)
     captureCommandAudio(cmdBuf);           // still on-device
     Intent cmd = commandModel(features(cmdBuf));   // OFFLINE recognition
     if (cmd.confidence > 0.6f)
       act(cmd.intent);                     // act locally; audio never leaves
   }

   float score = wakeWordModel(feats); // STAGE 1: tiny, ALWAYS-ONThe cheap wake-word model runs on every frame — always-on within the power budget because it is tiny.

   if (score < THRESHOLD) return; // keep listening cheaplyMost of the time nothing fires and the device just keeps listening cheaply — no heavy work, no cloud.

   // Wake word detected -> STAGE 2 runs only now (rarely)The heavier command recogniser runs only after a wake word, which is what makes always-on feasible on tiny hardware.

   act(cmd.intent); // act locally; audio never leavesCommands are recognised offline and acted on locally, so audio never leaves the device — privacy by design.

2. Tune false accepts and respond locally

   Tune the threshold for very few false accepts, keep the vocabulary small and offline, and drive local outputs for instant response.

Complete Source Code

The listing below is complete and compiles as written — there are no elided sections. Read the annotations under each block before you upload it.

#!/usr/bin/env python3
"""
Wake-Word Voice Assistant (on-device, offline)

Two-stage cascade: a tiny ALWAYS-ON wake-word model runs continuously;
only AFTER it fires does an offline command recogniser run. All audio
stays ON-DEVICE (privacy) with instant, network-independent response.
TinyML: tight compute/power; small vocabulary; tune LOW false accepts.
"""
THRESHOLD = 0.85                       # tuned for very low false accepts

class WakeWordAssistant:
    def __init__(self, mic, wake_model, command_model, actuator):
        self.mic = mic; self.wake = wake_model
        self.cmd = command_model; self.act = actuator

    def run(self):
        while True:
            frame = self.mic.read_frame()          # continuous, on-device
            feats = features(frame)                 # MFCC/log-mel

            # STAGE 1: cheap, always-on wake-word detection
            if self.wake.infer(feats) < THRESHOLD:
                continue                            # keep listening cheaply

            # STAGE 2: runs ONLY after the wake word (rarely) — offline
            audio = self.mic.capture_command()      # audio stays local
            cmd = self.cmd.infer(features(audio))   # offline command recognition
            if cmd.confidence > 0.6:
                self.act(cmd.intent)                # act locally + respond
            # nothing is ever sent to the cloud.

if __name__ == "__main__":
    a = WakeWordAssistant(Microphone(), WakeWordModel(),
                          CommandModel(), Actuator())
    a.run()
    # Private + responsive; small offline vocab; robustness limited by TinyML.

Configuration & Calibration

Configuration steps

• Configure the microphone, feature front-end and audio framing.
• Configure the wake-word model and detection threshold (false-accept-first).
• Configure the offline command recogniser and small vocabulary.
• Configure local outputs and privacy (audio stays on-device).

Calibration procedure

An uncalibrated sensor produces confident, precise, wrong numbers. Do this once per physical unit and record the constants.

1. Wake-word operating point

   Tune the threshold for very low false accepts and acceptable false rejects across noise conditions.

2. Command accuracy

   Validate offline command recognition; keep the vocabulary small and robust.

3. Resource fit

   Confirm models fit the device's compute/memory/power for always-on operation.

Dataset, Model & Training

Dataset

Wake-word training uses many recordings of the target phrase plus "not-the-word" negatives (speech, noise); command recognition uses labelled command audio.

Noise/accent coverage in the data determines real-world robustness on a small model.

Data preprocessing

• Extract audio features (e.g. MFCC/log-mel) on-device.
• Augment with noise/gain/accent variation for robustness.
• Frame audio for continuous streaming inference.

Hyperparameters

Training process

• Train the wake-word model on positives + varied negatives; heavily augment for noise/accents.
• Train/quantise a small command recogniser for offline use.
• Tune the operating point for low false accepts and acceptable false rejects.

Evaluation, Metrics & Deployment

The decisive metrics are false-accept rate (per hour) and false-reject rate for the wake word, plus offline command accuracy — all under tight compute/power budgets.

Figures from the reference training run described above — reproduce them before trusting your own changes.

Inference example

THRESHOLD = 0.85                       # tuned for low false accepts

def run(mic, wake_model, command_model, act):
    while True:
        frame = mic.read_frame()            # continuous audio, on-device
        feats = features(frame)             # MFCC/log-mel
        score = wake_model.infer(feats)     # STAGE 1: cheap, always-on

        if score >= THRESHOLD:              # wake word detected
            audio = mic.capture_command()   # STAGE 2 runs ONLY after wake
            cmd = command_model.infer(features(audio))   # offline recognition
            if cmd.confidence > 0.6:
                act(cmd.intent)             # act locally — audio never leaves
        # else: keep listening cheaply; nothing sent to the cloud.
    # TinyML: tiny model, small offline vocab; robustness limited by size.

Testing Procedure & Expected Output

Test from the bottom up. Confirm power, then each sensor in isolation, then the integrated loop — the first failing step tells you exactly where to look.

Bench-test checklist. If a row fails, stop and fix it before moving on.

Expected output

A private, offline voice assistant: reliable wake word, offline commands, local response, no cloud audio.

{
  "wake_word": "detected",
  "wake_score": 0.91,
  "command": "turn on the light",
  "handled": "offline",
  "audio_left_device": false
}

The wake word fired on-device, an offline command was recognised and acted on locally, and no audio ever left the device — private, responsive voice control.

Troubleshooting: Common Errors & Fixes

Performance Optimisation

• Keep the wake-word model tiny and always-on; run heavy work only after it fires.
• Tune hardest against false accepts.
• Keep commands offline with a small vocabulary.
• Quantise/fit models to the device's limits.
• Replace every delay() with a millis() comparison — blocking delays are the single most common cause of dropped readings.
• Sample sensors on a fixed cadence and publish on a slower one; you almost never need to transmit at the sampling rate.
• Move networking into its own FreeRTOS task so a slow DNS lookup cannot stall the control loop.
• Use uint8_t / uint16_t where the range allows; on an 8-bit AVR a 32-bit add costs four times as much.
• For battery builds use deep sleep between samples: an ESP32 drops from ~160 mA awake to about 10 µA asleep, which is the difference between days and months of runtime.
• Profile before optimising — print micros() deltas around each stage and fix the slowest one first.

Safety Precautions

• "Always listening" warrants genuine privacy care — keep audio on-device; do not stream it to the cloud.
• Be transparent that the device listens for a wake word, and give users control.
• A tiny model is limited in noise/accents — do not rely on it where misrecognition is dangerous.
• Secure any local storage of audio/commands.
• Wear eye protection when soldering or cutting, and solder in a ventilated space — rosin flux fumes are a respiratory irritant.
• Power the circuit through a bench supply with a current limit while you are testing. A 300 mA limit turns a wiring mistake into a beep instead of a dead board.
• Disconnect power before changing any wiring. Hot-plugging a sensor onto a live bus is the fastest way to lose a controller.

Maintenance

• Retrain/tune the wake word for new conditions and negatives.
• Update command models/vocabulary as needs change.
• Re-verify false-accept/reject rates over time.
• Keep audio local and privacy practices current.
• Re-check every screw terminal and header after the first week — thermal cycling loosens connections that felt tight on day one.
• Recalibrate at the interval given in the calibration section, and keep the constants in a text file next to the firmware — not only in flash.
• Keep a short logbook of firmware versions and what changed. Six months later you will not remember why that constant is 1.083.

Future Improvements & Upgrades

A working v1 is a platform, not a finish line. These are the upgrades that add the most capability for the least rework.

• Add on-device speech-to-text for a larger command set.
• Add speaker verification (only wake for enrolled users).
• Add noise suppression / beamforming for robustness.
• Add more capable offline models on stronger edge hardware.
• Design a proper PCB. Once the breadboard version has run for a month, moving to a two-layer board removes the intermittent-contact failures that dominate prototype faults.
• Add connectivity — an ESP32 and an MQTT publish turn a local gadget into something you can graph, alert on and analyse over months.
• Add persistent local storage (microSD or the on-chip flash) so a network outage does not create a hole in your data.
• Move configuration out of the source: a captive-portal setup page or a JSON config file makes the build reusable without a recompile.
• Add a battery and solar option so the unit survives a power cut and can be sited away from a socket.
• Write a small test harness that feeds synthetic sensor values through the decision logic, so you can validate thresholds without physically triggering the event.

Frequently Asked Questions

References & Learning Resources

These are the primary sources worth reading in full. Manufacturer datasheets always outrank forum posts when the two disagree.

1. Keyword / wake-word spottingReference
2. TinyML / on-device MLReference
3. Speech recognitionReference
4. MFCC audio featuresReference
5. TensorFlow Lite MicroDocs