Voice-Controlled Home Hub.

Project Overview

A wake-word voice assistant that runs entirely on a ₹900 microcontroller — no cloud, no account, no audio leaving the house — switching lights, fans and appliances from a spoken command.

Commercial voice assistants work by streaming a continuous audio buffer to a data centre. That is a reasonable engineering choice and an unreasonable privacy one. This project takes the opposite position: the microphone data never leaves the ESP32-S3. A small neural network trained on your own recordings runs on-device, and the only thing that ever reaches the network is a two-word MQTT message such as light/on.

That constraint drives every design decision. You cannot run a general speech recogniser in 512 KB of RAM, so the system is built as a keyword spotter — it recognises a fixed vocabulary of perhaps a dozen commands rather than transcribing arbitrary speech. In practice that is what a home hub actually needs. "Lights on", "fan off", "scene movie" covers the real usage; free-form dictation does not.

The signal chain is worth understanding because it is the same chain used in every production keyword spotter. Audio arrives from an INMP441 MEMS microphone over I²S at 16 kHz. A 30 ms sliding window is transformed to a Mel-frequency spectrogram — a compact time-frequency image that discards phase and most of the fine spectral detail humans do not use for phoneme identity. That image feeds a small depthwise-separable convolutional network, quantised to int8, which outputs a probability per keyword. The whole inference costs about 15 ms on the ESP32-S3, which has vector instructions specifically for this workload.

The ESP32-S3 rather than a plain ESP32 is a deliberate choice. The S3 adds 8 MB of PSRAM (enough for audio ring buffers and the model arena) and SIMD-like vector extensions that roughly triple int8 convolution throughput. On a classic ESP32 the same model runs at around 45 ms per inference, which is usable but leaves much less headroom for the audio front end.

Two failure modes dominate real deployments, and both are addressed here. False accepts — the hub switching the lights because the television said something similar — are handled with a confidence threshold plus a required run of consecutive positive frames. False rejects in a noisy kitchen are handled by training on your own room, with your own voice, including recordings of the background noise you actually have.

What this project does

• Continuously listens for a wake word entirely on-device, at about 0.4 W total power.
• Recognises a trained vocabulary of 8–12 command phrases with per-keyword confidence scores.
• Publishes the recognised intent as an MQTT message that Home Assistant, Node-RED or a relay board can act on.
• Switches four mains channels directly through an opto-isolated relay board when running standalone.
• Gives immediate audible and visual feedback so you always know whether a command landed.
• Logs every recognition — including rejected low-confidence ones — so you can tune the threshold from real data.
• Never transmits audio. The microphone buffer is overwritten in place and never written to flash or the network.

Real-World Applications

Deployment contexts where a build of this kind earns its keep.

Features & Capabilities

• Fully offline inference using TensorFlow Lite for Microcontrollers with an int8-quantised model under 60 KB.
• Mel-spectrogram front end computed with a fixed-point FFT, so feature extraction costs about 4 ms per window.
• Two-stage detection — a cheap always-on wake-word model gates a larger command model, cutting average power by roughly 60 %.
• Confidence hysteresis: three consecutive frames above threshold to accept, which nearly eliminates television false triggers.
• Per-keyword thresholds tunable at runtime over MQTT, because "off" is inherently harder to detect than "kitchen".
• Ring-buffered I²S capture on a dedicated FreeRTOS task pinned to core 0, so inference on core 1 never drops samples.
• Adaptive noise floor — the detector's energy gate tracks the room's ambient level over a 30 s window.
• WS2812 status ring giving a listening / thinking / accepted / rejected visual state.

Difficulty, Time & Required Skills

Skills you should have (or will pick up)

• Comfortable C++ including pointers, buffers and fixed-size arrays
• Basic digital signal processing intuition — sampling rate, windowing, spectrograms
• Enough Python to run a training notebook and read a confusion matrix
• Understanding of quantisation: why int8 and what it costs in accuracy
• FreeRTOS basics — tasks, cores, queues
• Mains wiring competence if you drive relays directly (or use a separate certified smart plug instead)

Bill of Materials

Every part below is commonly available from Indian and international hobby-electronics suppliers. Prices are indicative 2026 retail figures in Indian rupees and will drift — treat them as a budgeting guide, not a quotation.

Estimated total: ₹3,425, excluding tools, shipping and consumables.

Tools and consumables

• Soldering iron (temperature controlled, 350 °C) with 0.8 mm 60/40 or lead-free solder
• Digital multimeter — continuity, DC volts and current ranges
• Wire strippers, flush cutters and a small set of precision screwdrivers
• Heat-shrink tubing and a heat gun (or a lighter, carefully)
• A laptop with a USB port and the toolchain listed above

Hardware Specifications

Consolidated electrical and interface specifications for every active part in the build.

Power Budget & Supply Sizing

Add up the typical active current of every part, then size the supply with at least 50 % headroom so transmit bursts and motor inrush never brown out the controller.

Summed typical draw is 3559.4 mA. With a 1.5× design margin the supply should deliver at least 5400 mA continuously at the stated rail voltage.

Software Requirements & Development Environment

Reference toolchain: Arduino IDE 2.3.x with the ESP32 core 3.x (board: ESP32S3 Dev Module, PSRAM enabled) + Python 3.11 for training. Anything newer normally works; anything older may lack the board definitions used here.

• Install the Arduino IDE 2.3.x (or PlatformIO if you prefer a real editor and dependency locking).
• Add https://espressif.github.io/arduino-esp32/package_esp32_index.json under File → Preferences → Additional Board Manager URLs, then install esp32 from the Boards Manager.
• Set the correct port under Tools → Port. On Linux add yourself to the dialout group: sudo usermod -aG dialout $USER and log out and back in.
• Open the Serial Monitor at 115200 baud — every sketch here logs its state there.
• Keep File → Preferences → Show verbose output during: compilation switched on while you are debugging build errors.

Required libraries

Block Diagram

The block diagram shows the functional decomposition of the system — what senses, what decides, what acts, and where the data ends up.

Circuit Diagram & Wiring

Every signal line in the build is shown below, followed by a pin-by-pin connection table you can work through with a multimeter in hand.

Wire one row at a time and tick it off — most "it does not work" reports trace back to a single swapped pair.

Wiring explanation

• The INMP441 is an I²S digital microphone, not analogue. There is no ADC involved on the ESP32 side — the peripheral clocks 24-bit samples straight out of the microphone, which is why the noise floor is so much lower than an analogue electret plus op-amp.
• Tie the microphone L/R pin to ground to select the left channel. Left floating, the microphone may output on either slot and you will read silence half the time.
• Keep the three I²S lines short (under 15 cm) and run a ground wire alongside them. The bit clock is 1.024 MHz at 16 kHz × 32 bits × 2 channels and it radiates.
• Place the microphone port on the enclosure face with a 2–3 mm hole and a fabric mesh behind it. Do not cover it with anything solid, and do not glue it — mechanical coupling to the case turns every knock into a false trigger.
• The WS2812 ring wants 5 V data ideally, but works reliably from a 3.3 V ESP32-S3 for short runs. If the first pixel misbehaves, add a level shifter or sacrifice one pixel as a buffer.
• The relay board must be powered from the 5 V rail, not the ESP32 3V3 pin. Four energised coils draw about 280 mA — far beyond what the on-board regulator will supply.

System Architecture

Read the stack from the bottom up: physical hardware, the firmware that drives it, the transport that moves data off the device, and the software a human actually looks at.

Working Principle

Speech recognition on a microcontroller is an exercise in throwing information away intelligently. Raw audio at 16 kHz is 16 000 numbers per second; a keyword decision needs perhaps 2 000 numbers per second of useful information. The Mel spectrogram is the standard way of making that reduction, and it is worth understanding why it works.

Start with a 30 ms window of samples — 480 at 16 kHz. Speech is roughly stationary over that span: a vowel does not change identity in 30 ms. Multiply by a Hann window to taper the edges, because an abrupt cut produces spectral leakage that smears energy across every frequency bin. Take the magnitude of the FFT and discard the phase; for keyword identity, phase carries almost nothing. That leaves 256 magnitude bins.

Now compress those 256 bins into about 40, using triangular filters spaced on the Mel scale. The Mel scale is approximately linear below 1 kHz and logarithmic above, which mirrors how the cochlea resolves frequency — we discriminate 200 Hz from 300 Hz easily, and 5000 Hz from 5100 Hz not at all. Filters that follow that curve preserve the information the ear uses and throw away the rest. Take the logarithm of each filter output, because loudness perception is roughly logarithmic and because it compresses the dynamic range into something an int8 network can represent.

Slide that window forward 20 ms at a time and stack the results, and after one second you have a 49 × 40 image. That image is what the network actually sees. The word "kitchen" produces a visually distinctive pattern — a burst of high-frequency energy for the /k/, a formant structure for the vowel, another burst for the /tʃ/ — and a convolutional network is extremely good at learning those patterns.

The network itself is a depthwise-separable CNN, the architecture Google published as DS-CNN in their Hello Edge work and which remains the practical default for this task. A standard convolution over C input channels with K output channels and a 3 × 3 kernel costs 9·C·K multiply-accumulates per output pixel. A depthwise-separable convolution splits that into a 3 × 3 spatial filter per channel (9·C) followed by a 1 × 1 mix across channels (C·K), which is roughly 8–9× cheaper for typical channel counts at nearly the same accuracy. That factor is exactly what makes the difference between fitting in a microcontroller and not.

Finally, quantisation. Training happens in float32; deployment does not. Post-training quantisation maps each tensor to int8 using a per-axis scale and zero point, so a weight w is stored as round(w / scale) + zero_point. The model shrinks 4× and integer arithmetic runs several times faster on hardware with no FPU-heavy vector unit. Typical accuracy cost for this class of model is well under one percentage point, provided you supply a representative dataset during conversion so the converter can measure the real activation ranges.

The maths behind it

Mel scale conversion

m = 2595 · log10(1 + f / 700)
f = 700 · (10^(m / 2595) − 1)

40 filters spanning 80 Hz – 7600 Hz:
  m_low  = 2595 · log10(1 + 80/700)   ≈  120.4 mel
  m_high = 2595 · log10(1 + 7600/700) ≈ 2762.4 mel
  spacing = (2762.4 − 120.4) / 41     ≈   64.4 mel

Filter centres are evenly spaced in mel and therefore unevenly spaced in hertz — narrow and closely packed at low frequency, wide and sparse at high frequency.

Int8 quantisation

real  = scale × (quantised − zero_point)
scale = (real_max − real_min) / 255
zero_point = round(−real_min / scale) − 128

Example activation range [−6.0, +2.0]:
  scale      = 8.0 / 255      = 0.03137
  zero_point = round(6.0/0.03137) − 128 = 63

Every tensor carries its own scale and zero point, which is why a representative calibration dataset matters: the converter derives these constants from observed activations.

Inference cost

Standard 3×3 conv:  9 · C_in · C_out       MACs / pixel
Depthwise-separable: 9 · C_in + C_in · C_out

C_in = C_out = 64:
  standard    = 9 · 64 · 64 = 36 864 MACs
  separable   = 9 · 64 + 64 · 64 = 4 672 MACs
  reduction   = 7.9×

The saving grows with channel count, which is why every mobile and microcontroller vision or audio model uses this decomposition.

Program Flowchart

The firmware is a single cooperative loop. Nothing blocks for long, so networking, sensing and the user interface all stay responsive.

Assembly Instructions

Build on a breadboard first and only commit to solder once the whole system has run for an hour without a fault.

1. Bring up the microphone first

   Wire only the INMP441 and run the I²S capture sketch from step 1. Print the RMS of each buffer and confirm it rises when you speak and falls in silence. A microphone that reads a constant value has its L/R pin floating or its data line on the wrong GPIO.

2. Mount the microphone acoustically, not structurally

   Stick the INMP441 board to a small foam pad rather than directly to the enclosure. Line up the microphone port hole in the PCB with a 2.5 mm hole in the case and leave a 1 mm air gap. Cover the outside with acoustic mesh, never with tape.

3. Add the status ring and buzzer

   The WS2812 ring is the user interface. Fit it behind a diffuser — a disc of 1 mm white acrylic or even printer paper — so it reads as a glow rather than twelve point sources.

4. Wire the relay board on the far side of the enclosure

   Keep the mains section physically separated from the microphone and the ESP32-S3, with a solid barrier if the enclosure allows. Relay switching transients are broadband electrical noise, and the I²S clock lines are the last thing you want them coupling into.

5. Close up and re-test acoustically

   Recognition accuracy measured with the lid off is not the accuracy you will get with it on. The enclosure changes the frequency response. Always do your final threshold tuning with the case fully assembled and the unit in its final position.

Step-by-Step Implementation Guide

Work through these in order. Each step ends in something you can observe, so a failure is always localised to the step you just finished.

1. Capture audio over I²S and measure the level

   This sketch does nothing but read the microphone and print a level meter. Get it right before anything else — every later problem is easier to diagnose when you trust the audio.

   cpp01-i2s-capture.ino Copy Download

   #include <driver/i2s.h>
   
   #define I2S_BCLK  14
   #define I2S_LRCL  15
   #define I2S_DOUT  32
   #define SAMPLE_RATE 16000
   #define FRAME_LEN   512
   
   int32_t raw[FRAME_LEN];        // INMP441 delivers 24-bit left-justified in 32
   
   void setup() {
     Serial.begin(115200);
   
     i2s_config_t cfg = {
       .mode = (i2s_mode_t)(I2S_MODE_MASTER | I2S_MODE_RX),
       .sample_rate = SAMPLE_RATE,
       .bits_per_sample = I2S_BITS_PER_SAMPLE_32BIT,
       .channel_format = I2S_CHANNEL_FMT_ONLY_LEFT,
       .communication_format = I2S_COMM_FORMAT_STAND_I2S,
       .intr_alloc_flags = ESP_INTR_FLAG_LEVEL1,
       .dma_buf_count = 8,        // 8 x 512 samples = 256 ms of slack
       .dma_buf_len = FRAME_LEN,
       .use_apll = true           // cleaner clock than the PLL divider
     };
     i2s_pin_config_t pins = {
       .bck_io_num = I2S_BCLK,  .ws_io_num = I2S_LRCL,
       .data_out_num = I2S_PIN_NO_CHANGE, .data_in_num = I2S_DOUT
     };
     i2s_driver_install(I2S_NUM_0, &cfg, 0, NULL);
     i2s_set_pin(I2S_NUM_0, &pins);
   }
   
   void loop() {
     size_t got = 0;
     i2s_read(I2S_NUM_0, raw, sizeof(raw), &got, portMAX_DELAY);
     int n = got / sizeof(int32_t);
   
     // Shift right by 11 to land 24-bit audio in a signed 16-bit range.
     double sumSq = 0;
     for (int i = 0; i < n; i++) {
       int16_t s = (int16_t)(raw[i] >> 11);
       sumSq += (double)s * s;
     }
     float rms = sqrt(sumSq / n);
     float db  = 20.0f * log10f(rms / 32768.0f + 1e-9f);
   
     Serial.printf("rms %7.1f  %6.1f dBFS  ", rms, db);
     for (int i = 0; i < (int)((db + 80) / 2); i++) Serial.print('#');
     Serial.println();
   }

   Expand full listing (50 lines)
   I2S_BITS_PER_SAMPLE_32BITThe INMP441 is a 24-bit part but transmits in 32-bit slots. Configuring 16-bit here is the classic mistake — you get half of each sample and the audio sounds like static.

   raw[i] >> 11The 24 bits sit left-justified in the 32-bit word. Shifting right by 11 keeps the top 16 bits with a little headroom, which is a good working level for speech without clipping on a loud word.

   use_apll = trueThe audio PLL generates a much lower-jitter bit clock than dividing the main PLL. Jitter shows up as a raised noise floor, which directly costs you recognition accuracy in a quiet room.

   dma_buf_count = 8Eight buffers give 256 ms of slack. If inference occasionally overruns, DMA keeps filling buffers rather than dropping samples — dropped samples corrupt the spectrogram and produce mysterious mis-detections.

2. Collect your own training data

   Public keyword datasets are a starting point, not a solution. Record in the room the device will live in, with the people who will use it. Aim for 60–100 utterances per keyword across at least three speakers, plus five minutes of pure background noise from that room, plus a "not a command" class made of ordinary conversation and television audio.

   pythonrecord_samples.py Copy Download

   #!/usr/bin/env python3
   """Record labelled 1-second keyword clips at 16 kHz.
   
       python3 record_samples.py --label lights_on --count 60
   """
   import argparse
   import pathlib
   import queue
   import sys
   
   import numpy as np
   import sounddevice as sd
   import soundfile as sf
   
   RATE = 16_000
   CLIP = 1.0  # seconds — must match the model's input length
   
   
   def record_one(seconds: float = CLIP) -> np.ndarray:
       frames: queue.Queue = queue.Queue()
       with sd.InputStream(samplerate=RATE, channels=1, dtype="int16",
                           callback=lambda d, *_: frames.put(d.copy())):
           sd.sleep(int(seconds * 1000))
       chunks = []
       while not frames.empty():
           chunks.append(frames.get())
       clip = np.concatenate(chunks).flatten()
       # Pad or trim to exactly CLIP seconds so every example has one shape.
       want = int(RATE * seconds)
       return np.pad(clip, (0, max(0, want - len(clip))))[:want]
   
   
   def main() -> None:
       ap = argparse.ArgumentParser()
       ap.add_argument("--label", required=True)
       ap.add_argument("--count", type=int, default=60)
       ap.add_argument("--out", default="dataset")
       args = ap.parse_args()
   
       out = pathlib.Path(args.out) / args.label
       out.mkdir(parents=True, exist_ok=True)
       start = len(list(out.glob("*.wav")))
   
       print(f"Recording {args.count} clips for '{args.label}'.")
       print("Vary your distance, angle and speed. Enter to record, q to stop.")
       for i in range(args.count):
           if input(f"[{i + 1}/{args.count}] > ").strip().lower() == "q":
               break
           clip = record_one()
           peak = np.abs(clip).max() / 32768
           if peak < 0.02:
               print("  too quiet — discarded, move closer")
               continue
           if peak > 0.98:
               print("  clipped — discarded, move back")
               continue
           sf.write(out / f"{args.label}_{start + i:04d}.wav", clip, RATE)
           print(f"  saved  peak {peak:.2f}")
   
   
   if __name__ == "__main__":
       sys.exit(main())

   Expand full listing (62 lines)
   peak < 0.02 / > 0.98Automatic quality gates. Clips that are near-silent or clipped teach the model nothing useful and actively hurt — rejecting them at capture time is far cheaper than cleaning the dataset later.

   np.pad(...)[:want]Every example must be exactly one second. A model with a fixed input shape cannot accept variable-length audio, and silently truncating during training is a subtle source of label noise.

   "Vary your distance, angle and speed"This is the most important line in the script. A dataset recorded at one distance in one tone of voice produces a model that only works at that distance in that tone of voice.

3. Train and quantise the model

   The architecture is small enough to train on a laptop CPU in about fifteen minutes. Resist the temptation to make it bigger — accuracy on this task is bounded by your data, not your parameter count.

   pythontrain_kws.py Copy Download

   #!/usr/bin/env python3
   """Train a depthwise-separable CNN keyword spotter and export int8 TFLite."""
   import pathlib
   
   import numpy as np
   import tensorflow as tf
   
   RATE, CLIP = 16_000, 1.0
   N_MELS, N_FRAMES = 40, 49
   LABELS = ["_background", "_unknown", "lights_on", "lights_off",
             "fan_on", "fan_off", "scene_movie", "all_off"]
   
   
   def log_mel(waveform: tf.Tensor) -> tf.Tensor:
       """480-sample window, 320-sample hop -> 49 x 40 log-mel image."""
       stft = tf.signal.stft(waveform, frame_length=480, frame_step=320, fft_length=512)
       spec = tf.abs(stft)
       mel_w = tf.signal.linear_to_mel_weight_matrix(
           num_mel_bins=N_MELS, num_spectrogram_bins=stft.shape[-1],
           sample_rate=RATE, lower_edge_hertz=80.0, upper_edge_hertz=7600.0)
       mel = tf.tensordot(spec, mel_w, 1)
       return tf.math.log(mel + 1e-6)[..., tf.newaxis]
   
   
   def build_model() -> tf.keras.Model:
       inp = tf.keras.Input(shape=(N_FRAMES, N_MELS, 1))
       x = tf.keras.layers.Conv2D(32, (3, 3), strides=(2, 2), padding="same",
                                  use_bias=False)(inp)
       x = tf.keras.layers.BatchNormalization()(x)
       x = tf.keras.layers.ReLU()(x)
   
       for _ in range(4):                       # 4 depthwise-separable blocks
           x = tf.keras.layers.DepthwiseConv2D((3, 3), padding="same", use_bias=False)(x)
           x = tf.keras.layers.BatchNormalization()(x)
           x = tf.keras.layers.ReLU()(x)
           x = tf.keras.layers.Conv2D(32, (1, 1), padding="same", use_bias=False)(x)
           x = tf.keras.layers.BatchNormalization()(x)
           x = tf.keras.layers.ReLU()(x)
   
       x = tf.keras.layers.GlobalAveragePooling2D()(x)
       x = tf.keras.layers.Dropout(0.25)(x)
       out = tf.keras.layers.Dense(len(LABELS), activation="softmax")(x)
       return tf.keras.Model(inp, out)
   
   
   def load_dataset(root="dataset"):
       xs, ys = [], []
       for idx, label in enumerate(LABELS):
           for wav in pathlib.Path(root, label).glob("*.wav"):
               audio, _ = tf.audio.decode_wav(tf.io.read_file(str(wav)),
                                              desired_channels=1,
                                              desired_samples=int(RATE * CLIP))
               xs.append(log_mel(tf.squeeze(audio, -1)))
               ys.append(idx)
       return np.stack(xs), np.array(ys)
   
   
   def main() -> None:
       x, y = load_dataset()
       print(f"{len(x)} examples, {len(LABELS)} classes")
   
       perm = np.random.permutation(len(x))
       x, y = x[perm], y[perm]
       split = int(0.85 * len(x))
       x_tr, y_tr, x_va, y_va = x[:split], y[:split], x[split:], y[split:]
   
       model = build_model()
       model.compile(optimizer=tf.keras.optimizers.Adam(1e-3),
                     loss="sparse_categorical_crossentropy",
                     metrics=["accuracy"])
       model.fit(x_tr, y_tr, validation_data=(x_va, y_va),
                 epochs=60, batch_size=64,
                 callbacks=[tf.keras.callbacks.EarlyStopping(
                     patience=8, restore_best_weights=True)])
   
       # ---- int8 quantisation -------------------------------------------
       def representative():
           for i in range(min(300, len(x_tr))):
               yield [x_tr[i:i + 1].astype(np.float32)]
   
       conv = tf.lite.TFLiteConverter.from_keras_model(model)
       conv.optimizations = [tf.lite.Optimize.DEFAULT]
       conv.representative_dataset = representative
       conv.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
       conv.inference_input_type = tf.int8
       conv.inference_output_type = tf.int8
       tflite = conv.convert()
   
       pathlib.Path("kws_int8.tflite").write_bytes(tflite)
       print(f"model size: {len(tflite) / 1024:.1f} KB")
   
       # Emit a C array the sketch can include directly.
       with open("model_data.h", "w") as f:
           f.write("alignas(16) const unsigned char g_model[] = {\n")
           for i in range(0, len(tflite), 12):
               f.write("  " + ", ".join(f"0x{b:02x}" for b in tflite[i:i + 12]) + ",\n")
           f.write("};\nconst unsigned int g_model_len = %d;\n" % len(tflite))
   
   
   if __name__ == "__main__":
       main()

   Expand full listing (101 lines)
   log_mel()Feature extraction lives in the training script and is mirrored exactly in the firmware. Any mismatch — a different window length, a different mel range — silently destroys accuracy at deployment while training metrics still look perfect.

   DepthwiseConv2D + Conv2D 1×1This pair is the depthwise-separable block: spatial filtering per channel, then a pointwise mix across channels. It is what makes the model roughly eight times cheaper than plain convolutions.

   GlobalAveragePooling2DReplaces a flatten-plus-dense head. It removes tens of thousands of parameters and makes the model tolerant of small time shifts in the keyword.

   representative_datasetThe converter runs these samples through the float model to observe real activation ranges and pick per-tensor scales. Skip it and every activation gets a crude default range, which typically costs 5–15 points of accuracy.

   _background and _unknown classesTwo negative classes, not one. "_background" is room noise; "_unknown" is speech that is not a command. Merging them makes the model confuse silence with conversation and raises false accepts sharply.

4. Run inference on the device

   The firmware mirrors the training front end exactly, then runs the interpreter and applies hysteresis before acting.

   cpp02-inference-core.ino Copy Download

   #include <TensorFlowLite_ESP32.h>
   #include "tensorflow/lite/micro/micro_interpreter.h"
   #include "tensorflow/lite/micro/micro_mutable_op_resolver.h"
   #include "tensorflow/lite/schema/schema_generated.h"
   #include "model_data.h"
   
   constexpr int  N_FRAMES = 49, N_MELS = 40, N_LABELS = 8;
   constexpr int  ARENA_SIZE = 70 * 1024;
   
   static uint8_t *arena;                     // allocated in PSRAM
   static tflite::MicroInterpreter *interp;
   static TfLiteTensor *input, *output;
   
   const char *LABELS[N_LABELS] = {
     "_background", "_unknown", "lights_on", "lights_off",
     "fan_on", "fan_off", "scene_movie", "all_off"
   };
   // "off" words are acoustically weaker; they need a lower bar.
   const float THRESH[N_LABELS] = { 1.0f, 1.0f, 0.85f, 0.78f, 0.85f, 0.78f, 0.88f, 0.90f };
   
   void inferenceBegin() {
     arena = (uint8_t *)heap_caps_malloc(ARENA_SIZE, MALLOC_CAP_SPIRAM);
   
     static tflite::MicroMutableOpResolver<8> resolver;
     resolver.AddConv2D();
     resolver.AddDepthwiseConv2D();
     resolver.AddRelu();
     resolver.AddAveragePool2D();
     resolver.AddReshape();
     resolver.AddFullyConnected();
     resolver.AddSoftmax();
     resolver.AddQuantize();
   
     const tflite::Model *model = tflite::GetModel(g_model);
     static tflite::MicroInterpreter s(model, resolver, arena, ARENA_SIZE);
     interp = &s;
     interp->AllocateTensors();
     input  = interp->input(0);
     output = interp->output(0);
   
     Serial.printf("arena used: %u bytes\n", (unsigned)interp->arena_used_bytes());
   }
   
   // features[] holds the float log-mel image; quantise it into the tensor.
   int classify(const float *features, float *bestScore) {
     const float  s  = input->params.scale;
     const int    zp = input->params.zero_point;
     int8_t      *dst = input->data.int8;
   
     for (int i = 0; i < N_FRAMES * N_MELS; i++) {
       int v = (int)lroundf(features[i] / s) + zp;
       dst[i] = (int8_t)(v < -128 ? -128 : (v > 127 ? 127 : v));
     }
   
     if (interp->Invoke() != kTfLiteOk) return -1;
   
     const float os  = output->params.scale;
     const int   ozp = output->params.zero_point;
     int   best = 0;
     float bestP = -1;
     for (int i = 0; i < N_LABELS; i++) {
       float p = os * (output->data.int8[i] - ozp);
       if (p > bestP) { bestP = p; best = i; }
     }
     *bestScore = bestP;
     return best;
   }
   
   // Three consecutive agreeing frames above threshold before we act.
   int stableIntent(int label, float score) {
     static int   lastLabel = -1;
     static int   run = 0;
     static uint32_t lastFire = 0;
   
     if (label != lastLabel) { lastLabel = label; run = 0; }
     run = (score >= THRESH[label]) ? run + 1 : 0;
   
     if (run >= 3 && millis() - lastFire > 1500) {   // 1.5 s command debounce
       run = 0;
       lastFire = millis();
       return label;
     }
     return -1;
   }

   Expand full listing (84 lines)
   heap_caps_malloc(..., MALLOC_CAP_SPIRAM)The 70 KB tensor arena goes into the S3's external PSRAM rather than internal SRAM, leaving internal memory free for the I²S DMA buffers and the Wi-Fi stack — which are both latency-critical and must not be in PSRAM.

   MicroMutableOpResolver<8>Registers only the eight operators this model actually uses. The all-ops resolver pulls in every kernel TFLite Micro knows about and adds roughly 100 KB of flash for no benefit.

   lroundf(features[i] / s) + zpManual quantisation of the input. The scale and zero point come from the model file itself, so this code stays correct if you retrain and the ranges change.

   THRESH[] per labelUnvoiced fricatives such as the /f/ in "off" carry far less energy than a plosive, so a single global threshold either misses "off" or over-triggers on "on". Per-keyword thresholds are the single cheapest accuracy improvement available.

   run >= 3 && millis() - lastFire > 1500Two independent guards. The run counter rejects momentary spikes from a television; the debounce timer stops one long utterance from firing the same command three times.

Complete Source Code

The listing below is complete and compiles as written — there are no elided sections. Read the annotations under each block before you upload it.

/* ═══════════════════════════════════════════════════════════════
   Offline Voice-Controlled Home Hub — ESP32-S3 + INMP441 + TFLite Micro

   Audio never leaves the device. A dedicated FreeRTOS task on core 0
   fills a ring buffer from I²S; core 1 computes a log-mel spectrogram
   and runs an int8 keyword-spotting model. Only the resulting intent
   is published over MQTT.

   Board: ESP32S3 Dev Module, PSRAM: OPI, Flash: 8 MB
   ══════════════════════════════════════════════════════════════════ */

#include <WiFi.h>
#include <PubSubClient.h>
#include <ArduinoJson.h>
#include <driver/i2s.h>
#include <FastLED.h>
#include <math.h>

#include <TensorFlowLite_ESP32.h>
#include "tensorflow/lite/micro/micro_interpreter.h"
#include "tensorflow/lite/micro/micro_mutable_op_resolver.h"
#include "tensorflow/lite/schema/schema_generated.h"
#include "model_data.h"

/* ── configuration ──────────────────────────────────────────── */
#define WIFI_SSID   "YOUR_WIFI"
#define WIFI_PASS   "YOUR_PASSWORD"
#define MQTT_HOST   "192.168.1.50"
#define MQTT_PORT   1883
#define DEVICE_ID   "voice-hub"

#define I2S_BCLK    14
#define I2S_LRCL    15
#define I2S_DOUT    32
#define PIN_LEDS    48
#define PIN_BUZZER  17
#define N_LEDS      12

const uint8_t RELAY_PIN[4] = { 4, 5, 6, 7 };

#define SAMPLE_RATE 16000
#define WIN_LEN     480          // 30 ms
#define HOP_LEN     320          // 20 ms
#define FFT_LEN     512
#define N_MELS      40
#define N_FRAMES    49           // 49 hops ≈ 1.0 s
#define RING_LEN    (SAMPLE_RATE)   // 1 s of int16 audio
#define ARENA_SIZE  (70 * 1024)

const char *LABELS[] = { "_background", "_unknown", "lights_on", "lights_off",
                         "fan_on", "fan_off", "scene_movie", "all_off" };
const float THRESH[] = { 1.0f, 1.0f, 0.85f, 0.78f, 0.85f, 0.78f, 0.88f, 0.90f };
#define N_LABELS (sizeof(LABELS) / sizeof(LABELS[0]))

/* ── globals ────────────────────────────────────────────────── */
WiFiClient   net;
PubSubClient mqtt(net);
CRGB         leds[N_LEDS];

static int16_t  *ring;            // PSRAM audio ring buffer
static volatile uint32_t ringHead = 0;
static float    *melFilters;      // N_MELS x (FFT_LEN/2+1), precomputed
static float    *features;        // N_FRAMES x N_MELS
static float     noiseFloor = 0.002f;

static uint8_t  *arena;
static tflite::MicroInterpreter *interp;
static TfLiteTensor *inputT, *outputT;

/* ── mel filterbank (computed once at boot) ─────────────────── */
static float hzToMel(float hz)  { return 2595.0f * log10f(1.0f + hz / 700.0f); }
static float melToHz(float mel) { return 700.0f * (powf(10.0f, mel / 2595.0f) - 1.0f); }

void buildMelFilters() {
  const int bins = FFT_LEN / 2 + 1;
  melFilters = (float *)heap_caps_calloc(N_MELS * bins, sizeof(float), MALLOC_CAP_SPIRAM);

  float mLow = hzToMel(80.0f), mHigh = hzToMel(7600.0f);
  float edges[N_MELS + 2];
  for (int i = 0; i < N_MELS + 2; i++)
    edges[i] = melToHz(mLow + (mHigh - mLow) * i / (N_MELS + 1));

  for (int m = 0; m < N_MELS; m++) {
    float f0 = edges[m], f1 = edges[m + 1], f2 = edges[m + 2];
    for (int k = 0; k < bins; k++) {
      float f = (float)k * SAMPLE_RATE / FFT_LEN;
      float w = 0.0f;
      if (f >= f0 && f <= f1)      w = (f - f0) / (f1 - f0);
      else if (f > f1 && f <= f2)  w = (f2 - f) / (f2 - f1);
      melFilters[m * bins + k] = w;
    }
  }
}

/* ── radix-2 in-place FFT (real input, complex output) ──────── */
void fft(float *re, float *im, int n) {
  for (int i = 1, j = 0; i < n; i++) {          // bit-reversal permutation
    int bit = n >> 1;
    for (; j & bit; bit >>= 1) j ^= bit;
    j ^= bit;
    if (i < j) { float t = re[i]; re[i] = re[j]; re[j] = t;
                 t = im[i]; im[i] = im[j]; im[j] = t; }
  }
  for (int len = 2; len <= n; len <<= 1) {
    float ang = -2.0f * (float)M_PI / len;
    float wr = cosf(ang), wi = sinf(ang);
    for (int i = 0; i < n; i += len) {
      float cr = 1.0f, ci = 0.0f;
      for (int k = 0; k < len / 2; k++) {
        float ur = re[i + k],           ui = im[i + k];
        float vr = re[i + k + len / 2] * cr - im[i + k + len / 2] * ci;
        float vi = re[i + k + len / 2] * ci + im[i + k + len / 2] * cr;
        re[i + k] = ur + vr;            im[i + k] = ui + vi;
        re[i + k + len / 2] = ur - vr;  im[i + k + len / 2] = ui - vi;
        float nr = cr * wr - ci * wi;
        ci = cr * wi + ci * wr;         cr = nr;
      }
    }
  }
}

/* ── log-mel spectrogram over the last 1 s of the ring ──────── */
void computeFeatures() {
  static float re[FFT_LEN], im[FFT_LEN];
  const int bins = FFT_LEN / 2 + 1;
  uint32_t start = (ringHead + RING_LEN - (N_FRAMES - 1) * HOP_LEN - WIN_LEN) % RING_LEN;

  for (int f = 0; f < N_FRAMES; f++) {
    memset(re, 0, sizeof(re));
    memset(im, 0, sizeof(im));

    float prev = 0;
    for (int n = 0; n < WIN_LEN; n++) {
      float s = ring[(start + f * HOP_LEN + n) % RING_LEN] / 32768.0f;
      float pe = s - 0.97f * prev;                     // pre-emphasis
      prev = s;
      float w = 0.5f - 0.5f * cosf(2.0f * (float)M_PI * n / (WIN_LEN - 1)); // Hann
      re[n] = pe * w;
    }
    fft(re, im, FFT_LEN);

    for (int m = 0; m < N_MELS; m++) {
      float acc = 0;
      const float *row = &melFilters[m * bins];
      for (int k = 0; k < bins; k++)
        if (row[k] > 0) acc += row[k] * sqrtf(re[k] * re[k] + im[k] * im[k]);
      features[f * N_MELS + m] = logf(acc + 1e-6f);
    }
  }
}

/* ── model ──────────────────────────────────────────────────── */
void inferenceBegin() {
  arena = (uint8_t *)heap_caps_malloc(ARENA_SIZE, MALLOC_CAP_SPIRAM);

  static tflite::MicroMutableOpResolver<8> resolver;
  resolver.AddConv2D();       resolver.AddDepthwiseConv2D();
  resolver.AddRelu();         resolver.AddAveragePool2D();
  resolver.AddReshape();      resolver.AddFullyConnected();
  resolver.AddSoftmax();      resolver.AddQuantize();

  static tflite::MicroInterpreter s(tflite::GetModel(g_model), resolver,
                                    arena, ARENA_SIZE);
  interp = &s;
  interp->AllocateTensors();
  inputT  = interp->input(0);
  outputT = interp->output(0);
  Serial.printf("arena used %u B, model %u B\n",
                (unsigned)interp->arena_used_bytes(), g_model_len);
}

int classify(float *bestScore) {
  const float s = inputT->params.scale;
  const int  zp = inputT->params.zero_point;
  for (int i = 0; i < N_FRAMES * N_MELS; i++) {
    int v = (int)lroundf(features[i] / s) + zp;
    inputT->data.int8[i] = (int8_t)(v < -128 ? -128 : (v > 127 ? 127 : v));
  }
  if (interp->Invoke() != kTfLiteOk) return -1;

  const float os = outputT->params.scale;
  const int  ozp = outputT->params.zero_point;
  int best = 0; float bestP = -1;
  for (size_t i = 0; i < N_LABELS; i++) {
    float p = os * (outputT->data.int8[i] - ozp);
    if (p > bestP) { bestP = p; best = (int)i; }
  }
  *bestScore = bestP;
  return best;
}

/* ── audio capture task (core 0) ────────────────────────────── */
void audioTask(void *) {
  static int32_t raw[256];
  size_t got;
  for (;;) {
    i2s_read(I2S_NUM_0, raw, sizeof(raw), &got, portMAX_DELAY);
    int n = got / sizeof(int32_t);
    for (int i = 0; i < n; i++) {
      ring[ringHead] = (int16_t)(raw[i] >> 11);
      ringHead = (ringHead + 1) % RING_LEN;
    }
  }
}

float ringRms() {
  double acc = 0;
  for (int i = 0; i < 1600; i++) {                 // last 100 ms
    int16_t s = ring[(ringHead + RING_LEN - 1 - i) % RING_LEN];
    acc += (double)s * s;
  }
  return sqrtf(acc / 1600) / 32768.0f;
}

/* ── feedback ───────────────────────────────────────────────── */
void ledState(CRGB c, uint8_t brightness) {
  fill_solid(leds, N_LEDS, c);
  FastLED.setBrightness(brightness);
  FastLED.show();
}

/* ── actions ────────────────────────────────────────────────── */
void applyIntent(int label) {
  const char *name = LABELS[label];

  if      (!strcmp(name, "lights_on"))  digitalWrite(RELAY_PIN[0], LOW);
  else if (!strcmp(name, "lights_off")) digitalWrite(RELAY_PIN[0], HIGH);
  else if (!strcmp(name, "fan_on"))     digitalWrite(RELAY_PIN[1], LOW);
  else if (!strcmp(name, "fan_off"))    digitalWrite(RELAY_PIN[1], HIGH);
  else if (!strcmp(name, "scene_movie")) {
    digitalWrite(RELAY_PIN[0], HIGH);
    digitalWrite(RELAY_PIN[2], LOW);
  } else if (!strcmp(name, "all_off")) {
    for (int i = 0; i < 4; i++) digitalWrite(RELAY_PIN[i], HIGH);
  }

  JsonDocument doc;
  doc["device"] = DEVICE_ID;
  doc["intent"] = name;
  doc["ts"]     = millis() / 1000;
  char buf[128];
  size_t n = serializeJson(doc, buf, sizeof(buf));
  mqtt.publish("home/voice/" DEVICE_ID "/intent", (const uint8_t *)buf, n, false);

  ledState(CRGB::Green, 80);
  tone(PIN_BUZZER, 2400, 90);
  delay(220);
  ledState(CRGB::Blue, 12);
}

/* ── setup / loop ───────────────────────────────────────────── */
void setup() {
  Serial.begin(115200);
  for (int i = 0; i < 4; i++) { pinMode(RELAY_PIN[i], OUTPUT); digitalWrite(RELAY_PIN[i], HIGH); }
  pinMode(PIN_BUZZER, OUTPUT);

  FastLED.addLeds<WS2812B, PIN_LEDS, GRB>(leds, N_LEDS);
  ledState(CRGB::Orange, 30);

  ring     = (int16_t *)heap_caps_calloc(RING_LEN, sizeof(int16_t), MALLOC_CAP_SPIRAM);
  features = (float  *)heap_caps_calloc(N_FRAMES * N_MELS, sizeof(float), MALLOC_CAP_SPIRAM);
  buildMelFilters();

  i2s_config_t cfg = {
    .mode = (i2s_mode_t)(I2S_MODE_MASTER | I2S_MODE_RX),
    .sample_rate = SAMPLE_RATE,
    .bits_per_sample = I2S_BITS_PER_SAMPLE_32BIT,
    .channel_format = I2S_CHANNEL_FMT_ONLY_LEFT,
    .communication_format = I2S_COMM_FORMAT_STAND_I2S,
    .intr_alloc_flags = ESP_INTR_FLAG_LEVEL1,
    .dma_buf_count = 8, .dma_buf_len = 256, .use_apll = true
  };
  i2s_pin_config_t pins = { .bck_io_num = I2S_BCLK, .ws_io_num = I2S_LRCL,
                            .data_out_num = I2S_PIN_NO_CHANGE, .data_in_num = I2S_DOUT };
  i2s_driver_install(I2S_NUM_0, &cfg, 0, NULL);
  i2s_set_pin(I2S_NUM_0, &pins);

  inferenceBegin();

  WiFi.mode(WIFI_STA);
  WiFi.begin(WIFI_SSID, WIFI_PASS);
  for (int i = 0; i < 40 && WiFi.status() != WL_CONNECTED; i++) delay(250);
  mqtt.setServer(MQTT_HOST, MQTT_PORT);

  // Capture pinned to core 0; inference runs on core 1 in loop().
  xTaskCreatePinnedToCore(audioTask, "audio", 4096, NULL, 5, NULL, 0);

  ledState(CRGB::Blue, 12);
  Serial.println("Voice hub listening — audio stays on this device");
}

void loop() {
  if (!mqtt.connected() && WiFi.status() == WL_CONNECTED) mqtt.connect(DEVICE_ID);
  mqtt.loop();

  float rms = ringRms();
  noiseFloor = 0.999f * noiseFloor + 0.001f * rms;      // slow adaptation

  if (rms < noiseFloor * 3.0f) { delay(10); return; }   // energy gate

  ledState(CRGB::Cyan, 40);
  uint32_t t0 = micros();
  computeFeatures();
  uint32_t t1 = micros();
  float score;
  int label = classify(&score);
  uint32_t t2 = micros();

  static int lastLabel = -1, run = 0;
  static uint32_t lastFire = 0;
  if (label != lastLabel) { lastLabel = label; run = 0; }
  run = (label >= 2 && score >= THRESH[label]) ? run + 1 : 0;

  Serial.printf("%-12s %.2f  feat %lu us  inf %lu us  run %d\n",
                label >= 0 ? LABELS[label] : "?", score,
                (unsigned long)(t1 - t0), (unsigned long)(t2 - t1), run);

  if (run >= 3 && millis() - lastFire > 1500) {
    run = 0; lastFire = millis();
    applyIntent(label);
  } else {
    ledState(CRGB::Blue, 12);
  }
}

Configuration & Calibration

Configuration steps

• Set Tools → PSRAM → OPI PSRAM in the Arduino IDE. Without it, heap_caps_malloc(..., MALLOC_CAP_SPIRAM) returns null and the device crashes in setup().
• Set Tools → Partition Scheme → Huge APP (3 MB). The TFLite Micro runtime plus the model does not fit in the default 1.2 MB app partition.
• Set Tools → CPU Frequency → 240 MHz. At 160 MHz the feature extraction takes about 40 % longer, which eats the headroom that keeps the energy gate cheap.
• Regenerate model_data.h whenever you retrain, and update LABELS[] and THRESH[] to match the label order in train_kws.py. A mismatched label order produces a device that works confidently and wrongly.
• Tune THRESH[] from the serial log, not from intuition. Speak each command twenty times, note the scores, and set the threshold a little below the tenth percentile of true positives.
• Adjust the energy-gate multiplier (3.0) for your room. Higher misses quiet speech; lower burns power on every fridge compressor cycle.

Calibration procedure

An uncalibrated sensor produces confident, precise, wrong numbers. Do this once per physical unit and record the constants.

1. Establish the room noise floor

   Leave the device running with the serial log open for ten minutes with nobody speaking. Note the RMS values. The steady-state noiseFloor should settle within about 20 % of the median idle RMS. If it keeps climbing, something in the room is periodically loud and you should raise the gate multiplier.

2. Measure the true-positive score distribution

   Speak each command twenty times from the position you will normally use. Record the reported score for each. A well-trained model gives true positives clustered above 0.9 with occasional dips to 0.7. Set that keyword's threshold just below the lowest score you are willing to accept.

3. Measure the false-accept rate against television

   Play an hour of television or radio at normal volume with nobody in the room and count triggers. Zero is achievable. If you get more than one per hour, the fix is almost always more _unknown training data recorded from that same television, not a higher threshold.

4. Verify timing headroom

   The serial log prints feature and inference times. Feature extraction should be around 3–5 ms and inference around 12–18 ms on an ESP32-S3 at 240 MHz. If inference exceeds 40 ms, check that PSRAM is enabled and the CPU is at full clock.

Network Architecture & Connectivity

The networking here is deliberately thin. The device publishes an intent string and, optionally, subscribes to a threshold-tuning topic. That is all. Because no audio ever crosses the network, the security posture of the whole system is dramatically simpler than a cloud assistant's.

Communication protocol

Intents are published as JSON on home/voice/<device>/intent with QoS 0. QoS 0 is correct here: if a "lights on" message is lost, the user will simply say it again within two seconds, and a delayed duplicate arriving thirty seconds later would be worse than a loss.

The device also publishes a rolling score log on a separate topic at QoS 0. That stream is what you use to tune thresholds from real data rather than guesses, and it is cheap enough to leave on permanently.

Message contract between the device and the broker.

Cloud platform configuration

Nothing needs to leave your network. A Mosquitto broker and Home Assistant on the same Raspberry Pi is the complete backend. If you want remote access, expose Home Assistant through a reverse proxy with TLS rather than exposing the broker.

Dashboard setup

Feed the score topic into InfluxDB and plot score against label over time in Grafana. Two panels are enough: a scatter of accepted scores (which should cluster near 1.0) and a histogram of rejected scores (which tells you exactly how much threshold headroom you have).

Mobile app integration

Adding the device to Home Assistant as an MQTT sensor gives you the mobile app, history and automation engine without writing anything. An automation triggered on intent == "all_off" can then do far more than the four relays on the board — including telling you, via a phone notification, that it heard you.

Security considerations

• No audio is transmitted or stored. The ring buffer is overwritten continuously and never written to flash — verify this yourself before trusting the claim on any voice device, including this one.
• Use MQTT over TLS and per-device credentials if the broker is reachable from outside the LAN.
• Put the hub on an IoT VLAN. It has a microphone; treat it as the most sensitive device on the network even though it does not transmit audio.
• Enable ESP32-S3 flash encryption before deployment if physical access is a concern — the model and Wi-Fi credentials are both readable from an unencrypted flash image.
• Provide a hardware mute. A physical switch that cuts the microphone supply is the only mute a user can actually verify, and it costs ₹30.

Dataset, Model & Training

Dataset

The dataset is the project. Everything else is plumbing. You need three kinds of audio, and the ratio between them matters more than the total count.

Positives: 60–100 clips per command word, spread across every person who will use the device, at varied distances (0.5 m to 4 m), angles, speaking rates and volumes. Include deliberately sloppy pronunciations — that is what real usage sounds like.

Background: five to ten minutes of the actual room with nobody speaking, captured at different times of day so it includes the fridge, the fan, traffic and the air conditioner.

Unknown speech: at least as many clips as all your positives combined, drawn from ordinary conversation, television and radio. This is the class most people under-collect, and it is the direct cause of a hub that switches the lights during dinner.

Google's Speech Commands v0.02 dataset is a useful supplement for the _unknown class — 105 000 one-second clips of 35 words under a permissive licence — but it will not substitute for recordings of your own room.

Data preprocessing

• Resample everything to 16 kHz mono, 16-bit PCM. Mismatched sample rates between training and deployment are the most common silent failure in TinyML audio.
• Trim or pad every clip to exactly 1.000 s, aligning the keyword roughly centrally but with deliberate jitter of ±100 ms so the model does not learn a fixed onset time.
• Normalise each clip to a peak of about −3 dBFS, then apply random gain of ±6 dB during training. Peak normalisation alone teaches the model that loudness is a feature, which it should not be.
• Augment with time shift (±100 ms), background mixing at 0–15 dB SNR, and mild time stretching (0.9–1.1×). Do not augment with pitch shift beyond about ±10 % — it distorts formants and creates examples that do not occur in reality.
• Compute log-mel features with exactly the same parameters used in the firmware: 480-sample window, 320-sample hop, 512-point FFT, 40 mel bands from 80 Hz to 7600 Hz.

Model architecture

The network is a depthwise-separable CNN in the DS-CNN family. An initial strided 3×3 convolution reduces the 49 × 40 input to 25 × 20 with 32 channels, then four depthwise-separable blocks each apply a 3×3 spatial filter per channel followed by a 1×1 pointwise mix. Global average pooling collapses the spatial dimensions, dropout at 0.25 regularises, and a dense softmax produces eight class probabilities.

Every convolution uses use_bias=False because it is immediately followed by batch normalisation, which has its own shift parameter — a bias term there is redundant and simply adds parameters. At conversion time TFLite folds the batch-norm parameters into the preceding convolution weights, so the deployed model has no batch-norm layers at all.

Hyperparameters

Training process

• Split by speaker, not randomly. A random split puts clips of the same person saying the same word in both train and validation, which inflates validation accuracy by five to ten points and tells you nothing about how it will work for a guest.
• Watch the confusion matrix, not the accuracy number. On this task the interesting failures are always specific pairs — "lights on" versus "lights off" — and the fix is more data for that pair, not more epochs.
• Retrain after the first week of real use. Log every rejected utterance to the serial console, listen to the ones that should have worked, and add them to the dataset. Two rounds of this typically halves the false-reject rate.
• Evaluate the quantised model, not the float one. Run the TFLite interpreter over your validation set and compare — if int8 costs more than about two points, your representative dataset is not representative.

Evaluation, Metrics & Deployment

Figures below are from a reference run: eight classes, roughly 900 own recordings plus 4 000 Speech Commands clips for the unknown class, speaker-disjoint split, measured on an ESP32-S3 at 240 MHz with PSRAM enabled.

Figures from the reference training run described above — reproduce them before trusting your own changes.

Deployment

• Export the int8 flatbuffer as a C array with 16-byte alignment. Unaligned model data causes hard faults on some cores and, worse, silently wrong results on others.
• Register only the operators the model uses via MicroMutableOpResolver. The all-ops resolver adds roughly 100 KB of flash and no capability.
• Put the tensor arena in PSRAM and the DMA buffers in internal SRAM. Getting this backwards halves throughput because DMA cannot reach PSRAM efficiently.
• Print arena_used_bytes() once at boot and size the arena to that plus 30 %. Over-sizing wastes PSRAM; under-sizing fails at AllocateTensors() with an unhelpful message.
• Version the model. Embed a build hash in the header and publish it on the MQTT status topic, so you can tell which model a given device is running when you are debugging six of them.

Inference example

#!/usr/bin/env python3
"""Verify the quantised model on the host before flashing it."""
import numpy as np
import tensorflow as tf

interp = tf.lite.Interpreter(model_path="kws_int8.tflite")
interp.allocate_tensors()
inp, out = interp.get_input_details()[0], interp.get_output_details()[0]

in_scale, in_zp = inp["quantization"]
out_scale, out_zp = out["quantization"]
print(f"input  {inp['shape']} {inp['dtype'].__name__} scale={in_scale:.5f} zp={in_zp}")
print(f"output {out['shape']} {out['dtype'].__name__} scale={out_scale:.5f} zp={out_zp}")

# x_val holds float log-mel features produced by the same log_mel() used in training.
x_val = np.load("x_val.npy")
y_val = np.load("y_val.npy")

correct = 0
for x, y in zip(x_val, y_val):
    q = np.clip(np.round(x / in_scale) + in_zp, -128, 127).astype(np.int8)
    interp.set_tensor(inp["index"], q[np.newaxis, ...])
    interp.invoke()
    probs = out_scale * (interp.get_tensor(out["index"])[0].astype(np.int32) - out_zp)
    correct += int(np.argmax(probs) == y)

print(f"int8 accuracy: {correct / len(y_val):.4f}")

Testing Procedure & Expected Output

Test from the bottom up. Confirm power, then each sensor in isolation, then the integrated loop — the first failing step tells you exactly where to look.

Bench-test checklist. If a row fails, stop and fix it before moving on.

Expected output

The serial log during a successful recognition, showing the energy gate opening, the feature and inference timings, and the hysteresis counter reaching three:

arena used 47104 B, model 59392 B
Voice hub listening — audio stays on this device

_background  0.97  feat 4102 us  inf 15281 us  run 0
_unknown     0.81  feat 4098 us  inf 15266 us  run 0
lights_on    0.71  feat 4110 us  inf 15302 us  run 0
lights_on    0.94  feat 4104 us  inf 15288 us  run 1
lights_on    0.97  feat 4099 us  inf 15274 us  run 2
lights_on    0.96  feat 4107 us  inf 15291 us  run 3
  -> intent lights_on  relay 0 ON  published

_background  0.99  feat 4101 us  inf 15279 us  run 0
fan_off      0.66  feat 4106 us  inf 15285 us  run 0   (below threshold 0.78)
fan_off      0.83  feat 4103 us  inf 15277 us  run 1
fan_off      0.88  feat 4108 us  inf 15294 us  run 2
fan_off      0.91  feat 4100 us  inf 15281 us  run 3
  -> intent fan_off   relay 1 OFF published

Troubleshooting: Common Errors & Fixes

Performance Optimisation

• The energy gate is the single biggest power lever. Tuning it so the device runs inference on 5 % of frames rather than 100 % cuts average power by roughly six times.
• Precompute the mel filterbank and the Hann window at boot rather than per frame — recomputing cosf() 480 times per frame costs more than the FFT.
• Store the mel filterbank sparsely: most weights are zero, and skipping them (if (row[k] > 0)) roughly halves the mel projection cost.
• For battery operation, add a two-stage cascade — a tiny 8 KB wake-word model gating the full command model — and put the CPU in light sleep between energy-gate checks.
• Replace every delay() with a millis() comparison — blocking delays are the single most common cause of dropped readings.
• Sample sensors on a fixed cadence and publish on a slower one; you almost never need to transmit at the sampling rate.
• Move networking into its own FreeRTOS task so a slow DNS lookup cannot stall the control loop.
• Use uint8_t / uint16_t where the range allows; on an 8-bit AVR a 32-bit add costs four times as much.
• Batch several samples into one MQTT publish. Radio time, not CPU time, dominates the energy budget.
• Set the MQTT keep-alive to a value that matches your reporting interval so the broker does not churn reconnections.
• For battery builds use deep sleep between samples: an ESP32 drops from ~160 mA awake to about 10 µA asleep, which is the difference between days and months of runtime.

Safety Precautions

• A voice-controlled relay can switch a load while nobody is watching. Never put a heater, an iron or anything with a thermal runaway mode on a voice-controlled channel without an independent thermal cut-out.
• Fit a physical microphone mute switch. Software mute is not verifiable by the user, and a device with an unverifiable microphone in a bedroom is a reasonable thing for people to object to.
• Mains voltage kills. Anything on the load side of the relay is at 230 V. Do not work on a powered circuit, and never leave exposed mains wiring on a bench where someone could touch it.
• Keep at least 6 mm of creepage between the mains and low-voltage sides of any board you make, and never route mains tracks under the microcontroller.
• Have a qualified electrician do the final installation into a consumer unit or wall fitting. In most jurisdictions this is a legal requirement, not a suggestion.
• Fit an RCD/RCBO upstream and fuse the load appropriately for its rating.
• Wear eye protection when soldering or cutting, and solder in a ventilated space — rosin flux fumes are a respiratory irritant.
• Power the circuit through a bench supply with a current limit while you are testing. A 300 mA limit turns a wiring mistake into a beep instead of a dead board.
• Disconnect power before changing any wiring. Hot-plugging a sensor onto a live bus is the fastest way to lose a controller.

Maintenance

• Re-check every screw terminal and header after the first week — thermal cycling loosens connections that felt tight on day one.
• Keep the broker and dashboard containers patched, and rotate device credentials at least once a year.
• Recalibrate at the interval given in the calibration section, and keep the constants in a text file next to the firmware — not only in flash.
• Keep a short logbook of firmware versions and what changed. Six months later you will not remember why that constant is 1.083.

Future Improvements & Upgrades

A working v1 is a platform, not a finish line. These are the upgrades that add the most capability for the least rework.

• Add a second microphone and simple delay-and-sum beamforming. Two INMP441s 60 mm apart give roughly 4–6 dB of directional gain, which is worth more in a noisy kitchen than any model improvement.
• Implement two-stage cascade detection — a tiny always-on wake word gating the full command model — for a large reduction in average power.
• Add speaker verification so the hub only accepts commands from enrolled household voices. A small embedding model plus cosine similarity is enough for a home threat model.
• Add on-device text-to-speech confirmation using a small concatenative engine, so the hub can say "lights on" instead of beeping.
• Support OTA model updates so retraining does not require a USB cable.
• Design a proper PCB. Once the breadboard version has run for a month, moving to a two-layer board removes the intermittent-contact failures that dominate prototype faults.
• Add over-the-air firmware updates so you never have to physically reach a deployed node again.
• Add persistent local storage (microSD or the on-chip flash) so a network outage does not create a hole in your data.
• Move configuration out of the source: a captive-portal setup page or a JSON config file makes the build reusable without a recompile.
• Add a battery and solar option so the unit survives a power cut and can be sited away from a socket.
• Write a small test harness that feeds synthetic sensor values through the decision logic, so you can validate thresholds without physically triggering the event.

Frequently Asked Questions

References & Learning Resources

These are the primary sources worth reading in full. Manufacturer datasheets always outrank forum posts when the two disagree.

1. Zhang et al., "Hello Edge: Keyword Spotting on Microcontrollers" (DS-CNN architecture)arXiv:1711.07128
2. Warden, "Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition"arXiv:1804.03209
3. TensorFlow Lite for Microcontrollers — official guideGoogle AI Edge
4. Post-training integer quantisation — TensorFlow documentationTensorFlow
5. INMP441 omnidirectional MEMS microphone with I²S — datasheetTDK InvenSense
6. ESP32-S3 Technical Reference Manual — I²S peripheral and vector instructionsEspressif Systems
7. Davis & Mermelstein, "Comparison of parametric representations for monosyllabic word recognition" (origin of MFCC)IEEE TASSP, 1980
8. Edge Impulse — keyword spotting tutorialEdge Impulse