Speech-to-Text Transcriber.

Project Overview

Real-time transcription with speaker diarization — converting speech to text and labelling who said what across a meeting.

Turning speech into text — automatic speech recognition (ASR) — underpins captions, voice notes, meeting records and accessibility, and modern models have made it remarkably good. This project builds a transcriber that goes beyond raw transcription to answer the question a meeting record actually needs: not just what was said, but who said it. It combines real-time transcription with speaker diarization — segmenting the audio by speaker so the transcript reads "Alice: … / Bob: …" rather than an undifferentiated wall of text.

Two capabilities work together. Transcription (ASR) converts the audio to words, ideally in real time (streaming) so captions appear as people speak. Diarization answers "who spoke when" by detecting speaker changes and clustering the speech into distinct speakers — typically by turning each snippet of voice into a voice embedding and grouping similar-sounding segments together. Fused, they produce a speaker-attributed transcript. The system also handles the practicalities of real audio: streaming for low latency, and robustness to the messiness of real meetings.

The value is usable meeting records, searchable and attributed, plus captions and accessibility. It is honest about the genuinely hard parts of real-world speech: overlapping speech (people talking over each other defeats diarization), background noise, accents and domain vocabulary, poor microphones, and knowing how many speakers there are; accuracy (word error rate) and diarization quality both degrade in the wild, and the streaming-vs-accuracy trade-off is real. There are also privacy obligations — recording and transcribing people's speech needs consent and care. Built honestly — strong ASR, embedding-based diarization, streaming, and realistic about noise and overlap — it is both a genuinely useful transcription tool and a rich lesson in combining two speech-AI capabilities into a practical system.

What this project does

• Transcribes speech to text in real time
• Diarizes — labels who said what
• Segments and clusters audio by speaker (voice embeddings)
• Produces speaker-attributed transcripts
• Streams for low-latency captions
• Supports meeting records, captions, accessibility
• Handles (imperfectly) noise, accents and overlap

Real-World Applications

Deployment contexts where a build of this kind earns its keep.

Features & Capabilities

• ASR (speech-to-text), streaming
• Speaker diarization (who spoke when)
• Voice-embedding speaker clustering
• Speaker-attributed transcripts
• Real-time / batch modes
• Robustness handling (noise/accents)
• Honest about overlap, noise and privacy

Difficulty, Time & Required Skills

Skills you should have (or will pick up)

• Automatic speech recognition (streaming)
• Speaker diarization and voice embeddings
• Clustering speech segments by speaker
• Fusing ASR + diarization into transcripts
• Handling noise/overlap and privacy

Bill of Materials

Every part below is commonly available from Indian and international hobby-electronics suppliers. Prices are indicative 2026 retail figures in Indian rupees and will drift — treat them as a budgeting guide, not a quotation.

Estimated total: ₹0, excluding tools, shipping and consumables.

Tools and consumables

• Soldering iron (temperature controlled, 350 °C) with 0.8 mm 60/40 or lead-free solder
• Digital multimeter — continuity, DC volts and current ranges
• Wire strippers, flush cutters and a small set of precision screwdrivers
• Heat-shrink tubing and a heat gun (or a lighter, carefully)
• A laptop with a USB port and the toolchain listed above

Hardware Specifications

This is a pure-software speech system — no electronic hardware beyond a microphone to specify. The "platform" is a computer plus ASR/diarization models: a GPU (local models) or a hosted ASR API handles recognition, and a CPU handles diarization clustering and streaming.

Audio-capture quality (mic, room) strongly affects accuracy. Memory scales with audio length and model size; a deployment adds capture, a transcript store and consent/privacy tooling. Everything else is the software stack, models and libraries below.

Software Requirements & Development Environment

Reference toolchain: Python 3.11 + ASR / diarization. Anything newer normally works; anything older may lack the board definitions used here.

• Install the Arduino IDE 2.3.x (or PlatformIO if you prefer a real editor and dependency locking).
• Add https://espressif.github.io/arduino-esp32/package_esp32_index.json under File → Preferences → Additional Board Manager URLs, then install esp32 from the Boards Manager.
• Set the correct port under Tools → Port. On Linux add yourself to the dialout group: sudo usermod -aG dialout $USER and log out and back in.
• Open the Serial Monitor at 115200 baud — every sketch here logs its state there.
• Keep File → Preferences → Show verbose output during: compilation switched on while you are debugging build errors.

Required libraries

Block Diagram

The block diagram shows the functional decomposition of the system — what senses, what decides, what acts, and where the data ends up.

Circuit Diagram & Wiring

The "wiring" is the transcription data flow — meeting audio is transcribed by ASR and, in parallel, diarized into speakers; the two are fused into a speaker-attributed transcript, streamed in real time.

Wire one row at a time and tick it off — most "it does not work" reports trace back to a single swapped pair.

Wiring explanation

• Capture meeting audio (good microphones help a lot).
• Transcribe with ASR (streaming for real-time captions).
• Diarize: segment and cluster by speaker (voice embeddings).
• Fuse ASR text with speaker segments into an attributed transcript.
• Handle noise/overlap as best possible; obtain consent to record.

System Architecture

Read the stack from the bottom up: physical hardware, the firmware that drives it, the transport that moves data off the device, and the software a human actually looks at.

Working Principle

Raw transcription answers "what was said"; a useful meeting record answers "who said what". A wall of unattributed text is far less usable than a transcript that reads "Alice: … / Bob: …", because attribution is what makes a record searchable, followable and actionable. So this project deliberately combines two distinct speech-AI capabilities — recognition and diarization — because each alone is insufficient for the goal.

Automatic speech recognition converts audio to words, and modern models do this well. The important practical dimension is streaming: for live captions the system must transcribe as people speak, emitting text with low latency, which trades a little accuracy against immediacy (a batch model that sees the whole utterance is generally more accurate than one forced to commit word-by-word). ASR quality is measured by word error rate, and it degrades with noise, accents and unfamiliar vocabulary.

Speaker diarization is the "who spoke when" half, and it works quite differently from recognition. Rather than understanding words, it analyses voice characteristics: the audio is segmented at speaker changes, each segment is turned into a voice embedding (a vector capturing that voice's timbre), and segments are clustered so that all the segments from one person group together and get a consistent speaker label. Diarization does not need to know who the speakers are by name — it separates them into "Speaker 1, Speaker 2, …" by voice similarity. Fusing the two — aligning ASR words in time with diarization's speaker segments — produces the attributed transcript that is the whole point.

The honesty this project requires is that real-world speech is messy in ways that specifically break these systems. The hardest is overlapping speech: when two people talk at once, diarization (and ASR) struggle badly, because the "who is speaking" signal is genuinely ambiguous — and meetings are full of interruptions and cross-talk. Background noise, poor microphones, strong accents and domain-specific vocabulary all raise word error rate; and knowing how many speakers there are (or handling people joining/leaving) is itself hard. So both word accuracy and diarization quality degrade in the wild, and the streaming-vs-accuracy tension is real — a system tuned for live captions is not the same as one tuned for a perfect after-the-fact transcript. There is also a serious privacy dimension: recording and transcribing people's speech is capturing personal data and generally requires consent and careful, secure handling. Built with strong streaming ASR, embedding-based diarization, and clear eyes about noise, overlap and privacy, the transcriber delivers real value — attributed, searchable records and live captions — while teaching how to combine two speech capabilities into a practical, honestly-bounded system.

The maths behind it

Recognition (ASR)

ASR(audio) → words (+ timings)

Streaming: emit text with low latency (trades some accuracy).
Quality = word error rate (WER); worse with noise/accents.

Diarization (who spoke when)

segment audio at speaker changes
for each segment: v = voice_embedding(segment)
cluster {v} → speaker labels (Speaker 1, 2, ...)

Separates speakers by VOICE similarity, not by name.

Fusion → attributed transcript

align ASR words (by time) with speaker segments
→ "Alice: ...", "Bob: ..."

Overlapping speech breaks this — the who-is-speaking signal
becomes ambiguous when people talk at once.

Program Flowchart

The firmware is a single cooperative loop. Nothing blocks for long, so networking, sensing and the user interface all stay responsive.

Assembly Instructions

Build on a breadboard first and only commit to solder once the whole system has run for an hour without a fault.

1. Set up streaming ASR

   Capture audio and transcribe it with a streaming ASR model for low-latency text.

2. Add speaker diarization

   Segment the audio, embed each segment's voice, and cluster segments into speakers.

3. Fuse into an attributed transcript

   Align ASR words with speaker segments to produce "who said what", handling overlap as best possible.

Step-by-Step Implementation Guide

Work through these in order. Each step ends in something you can observe, so a failure is always localised to the step you just finished.

1. Transcribe, diarize and attribute

   Run ASR for words and diarization for speakers, then align words to speakers into an attributed transcript.

   pythontranscribe.py Copy Download

   def transcribe(audio, asr, embed, cluster):
       words = asr.stream(audio)                  # what was said (streaming)
   
       segments = segment_by_turns(audio)         # who-when: split at changes
       speakers = cluster([embed(s) for s in segments])   # cluster voice embeddings
   
       out = []
       for w in words:                            # fuse words with speakers
           spk = speaker_at(w.time, segments, speakers)
           out.append({"speaker": spk, "text": w.text})
       return out                                 # "Alice: ...", "Bob: ..."

   words = asr.stream(audio) # what was said (streaming)Streaming ASR emits words with low latency for live captions — the recognition half.

   speakers = cluster([embed(s) for s in segments]) # cluster voice embeddingsDiarization turns each segment into a voice embedding and clusters by voice similarity, separating speakers without knowing their names.

   spk = speaker_at(w.time, segments, speakers)Fusion aligns each word in time with the speaker who was talking then — producing attribution.

   return out # "Alice: ...", "Bob: ..."The result is a speaker-attributed transcript — the usable meeting record that is the goal.

2. Handle overlap, noise and consent

   Attribute overlapping speech on a best-effort basis, mitigate noise, and ensure recording consent and secure handling.

Complete Source Code

The listing below is complete and compiles as written — there are no elided sections. Read the annotations under each block before you upload it.

#!/usr/bin/env python3
"""
Speech-to-Text Transcriber with Diarization

Combines streaming ASR (what was said) with speaker DIARIZATION (who
said it) — voice-embedding clustering — into a speaker-attributed
transcript. Overlapping speech, noise and accents degrade it; recording
speech is personal data requiring consent and secure handling.
"""
class Transcriber:
    def __init__(self, asr, embedder, cluster):
        self.asr = asr; self.embed = embedder; self.cluster = cluster

    def transcribe(self, audio, stream=True):
        # 1) ASR — words with timings (streaming for live captions)
        words = self.asr.stream(audio) if stream else self.asr.batch(audio)

        # 2) Diarization — who spoke when
        segments = segment_by_turns(audio)             # split at speaker changes
        embs = [self.embed(s.audio) for s in segments]  # voice embeddings
        labels = self.cluster(embs)                     # cluster by voice

        # 3) Fuse — align words to the speaker talking at that time
        transcript = []
        for w in words:
            spk = label_at(w.time, segments, labels)    # best-effort in overlap
            transcript.append({"speaker": spk, "text": w.text, "t": round(w.time, 2)})
        return transcript                               # "Alice: ...", "Bob: ..."

if __name__ == "__main__":
    tr = Transcriber(StreamingASR(), SpeakerEmbedder(), cluster_speakers)
    for line in tr.transcribe(meeting_audio()):
        print(f'{line["speaker"]}: {line["text"]}')
    # Overlap/noise/accents degrade quality; obtain consent to record.

Configuration & Calibration

Configuration steps

• Configure the ASR model and streaming latency.
• Configure diarization (segmentation, embeddings, clustering, speaker count).
• Configure fusion/attribution and output format.
• Configure consent, secure storage and retention.

Calibration procedure

An uncalibrated sensor produces confident, precise, wrong numbers. Do this once per physical unit and record the constants.

1. ASR accuracy

   Measure WER on representative audio; adapt vocabulary for the domain.

2. Diarization

   Tune segmentation/clustering and speaker count; evaluate on multi-speaker audio.

3. Latency vs accuracy

   Balance streaming latency against accuracy for the use case.

Dataset, Model & Training

Dataset

ASR and diarization typically use pretrained models; the "data" at use time is the meeting audio. Domain vocabulary and speaker/voice variety affect quality.

Noise, accent and overlap coverage in the models' training data shape real-world robustness.

Data preprocessing

• Capture/clean audio; segment into frames for streaming.
• Voice-activity detection; segment at speaker changes for diarization.
• Extract features/embeddings for ASR and speaker clustering.

Hyperparameters

Training process

• Use pretrained ASR and speaker-embedding models; tune thresholds/clustering.
• Optionally adapt vocabulary for the domain; evaluate on realistic (noisy/overlapping) audio.
• Balance streaming latency against accuracy for the use case.

Evaluation, Metrics & Deployment

ASR is measured by word error rate and diarization by diarization error rate — both degrade with noise and, especially, overlapping speech.

Figures from the reference training run described above — reproduce them before trusting your own changes.

Inference example

def transcribe_meeting(audio, asr, embed, cluster):
    words = asr.stream(audio)                  # ASR: what was said (streaming)

    # Diarization: who spoke when
    segments = segment_by_turns(audio)         # split at speaker changes
    embs = [embed(s) for s in segments]        # voice embeddings
    speakers = cluster(embs)                   # group segments by voice

    # Fuse: align words with speaker segments
    transcript = []
    for w in words:
        spk = speaker_at(w.time, segments, speakers)   # who was speaking then
        transcript.append({"speaker": spk, "text": w.text, "t": w.time})
    return transcript                          # "Alice: ...", "Bob: ..."
    # Overlapping speech, noise and accents degrade this; get consent to record.

Testing Procedure & Expected Output

Test from the bottom up. Confirm power, then each sensor in isolation, then the integrated loop — the first failing step tells you exactly where to look.

Bench-test checklist. If a row fails, stop and fix it before moving on.

Expected output

A speaker-attributed transcript (and live captions) from meeting audio.

[
  { "speaker": "Speaker 1", "text": "Let's start with the budget.", "t": 0.4 },
  { "speaker": "Speaker 2", "text": "Sure, revenue is up this quarter.", "t": 3.1 },
  { "speaker": "Speaker 1", "text": "Good — any risks?", "t": 6.0 }
]

A speaker-attributed transcript — who said what, with timings — far more usable than an undifferentiated wall of text; overlapping speech would have degraded the attribution.

Troubleshooting: Common Errors & Fixes

Performance Optimisation

• Use streaming ASR for live captions; batch for max accuracy.
• Diarize with voice embeddings + clustering; set/estimate speaker count.
• Improve capture (mics) — it beats post-processing for noise/overlap.
• Balance latency vs accuracy for the use case.
• Replace every delay() with a millis() comparison — blocking delays are the single most common cause of dropped readings.
• Sample sensors on a fixed cadence and publish on a slower one; you almost never need to transmit at the sampling rate.
• Move networking into its own FreeRTOS task so a slow DNS lookup cannot stall the control loop.
• Use uint8_t / uint16_t where the range allows; on an 8-bit AVR a 32-bit add costs four times as much.
• Profile before optimising — print micros() deltas around each stage and fix the slowest one first.

Safety Precautions

• Recording and transcribing speech is personal data — obtain consent and handle it securely with retention limits.
• Accuracy degrades with noise, accents and overlap — do not treat transcripts as perfect records.
• Be transparent that a meeting is being transcribed.
• Secure transcripts and audio against unauthorised access.
• Wear eye protection when soldering or cutting, and solder in a ventilated space — rosin flux fumes are a respiratory irritant.
• Power the circuit through a bench supply with a current limit while you are testing. A 300 mA limit turns a wiring mistake into a beep instead of a dead board.
• Disconnect power before changing any wiring. Hot-plugging a sensor onto a live bus is the fastest way to lose a controller.

Maintenance

• Update ASR/diarization models as they improve.
• Adapt vocabulary for changing domains.
• Re-evaluate WER/diarization on real audio.
• Review consent and data-handling practices.
• Re-check every screw terminal and header after the first week — thermal cycling loosens connections that felt tight on day one.
• Recalibrate at the interval given in the calibration section, and keep the constants in a text file next to the firmware — not only in flash.
• Keep a short logbook of firmware versions and what changed. Six months later you will not remember why that constant is 1.083.

Future Improvements & Upgrades

A working v1 is a platform, not a finish line. These are the upgrades that add the most capability for the least rework.

• Add speaker identification (names, with enrolment/consent).
• Add overlap-aware diarization/separation.
• Add punctuation, summaries and action-item extraction.
• Add multilingual transcription.
• Design a proper PCB. Once the breadboard version has run for a month, moving to a two-layer board removes the intermittent-contact failures that dominate prototype faults.
• Add connectivity — an ESP32 and an MQTT publish turn a local gadget into something you can graph, alert on and analyse over months.
• Add persistent local storage (microSD or the on-chip flash) so a network outage does not create a hole in your data.
• Move configuration out of the source: a captive-portal setup page or a JSON config file makes the build reusable without a recompile.
• Add a battery and solar option so the unit survives a power cut and can be sited away from a socket.
• Write a small test harness that feeds synthetic sensor values through the decision logic, so you can validate thresholds without physically triggering the event.

Frequently Asked Questions

References & Learning Resources

These are the primary sources worth reading in full. Manufacturer datasheets always outrank forum posts when the two disagree.

1. Speech recognitionReference
2. Speaker diarisationReference
3. Speaker embeddingsReference
4. Word error rateReference
5. Voice activity detectionReference