Handwriting & OCR Engine.

Project Overview

Recognises handwritten digits and text and turns pages of writing into editable text — from the classic MNIST digit to full-page OCR.

Turning an image of writing into editable, searchable text — optical character recognition (OCR) — is one of the oldest and most useful applications of machine vision, digitising forms, notes, historical documents and mail. This project builds an OCR engine that spans the whole ladder: from recognising a single handwritten digit (the famous MNIST problem that launched a thousand ML careers) up to reading lines and pages of handwritten or printed text and outputting them as editable text. It is the canonical way to learn image classification and then see how real OCR composes that skill into a document pipeline.

The foundation is classifying a single character. MNIST — 28×28 images of handwritten digits 0–9 — is the "hello world" of deep learning: a small convolutional neural network learns to map each image to its digit with very high accuracy, teaching the core mechanics of training a classifier. But real text is not pre-segmented single characters, so full OCR adds the surrounding pipeline: detecting and segmenting lines, words and characters from a page, recognising each (single characters, or whole sequences with a sequence model that avoids brittle per-character cutting), and post-processing with a dictionary/language model to fix errors (turning "recognise" from "recogmse").

The value is a working document-digitisation tool and a complete, layered lesson: single-character classification, then sequence recognition, then a full page pipeline. It is honest that handwriting is much harder than print (enormous variation between and within writers, cursive, messy layouts), that engine accuracy depends heavily on image quality (resolution, contrast, skew), and that a benchmark digit-classifier is a long way from robust page OCR. Built honestly — mastering the MNIST core, then composing detection, sequence recognition and language post-processing — it delivers both a genuinely useful OCR engine and the clearest possible progression from a toy classifier to a real applied-vision system.

What this project does

• Recognises handwritten digits (MNIST) with a CNN
• Reads lines and pages of handwritten/printed text
• Segments text into lines/words/characters
• Recognises character sequences (avoiding brittle cutting)
• Post-processes with a dictionary/language model
• Outputs editable, searchable text from images
• Progresses from single-character to full-page OCR

Real-World Applications

Deployment contexts where a build of this kind earns its keep.

Features & Capabilities

• MNIST digit classification (CNN core)
• Text detection/segmentation
• Sequence recognition (line/word)
• Language-model post-processing
• Handwriting and print support
• Editable text output
• Honest about handwriting difficulty and image quality

Difficulty, Time & Required Skills

Skills you should have (or will pick up)

• CNN image classification (MNIST)
• Text detection and segmentation
• Sequence recognition (CTC/seq2seq)
• Language-model post-processing
• OCR pipeline composition and evaluation

Bill of Materials

Every part below is commonly available from Indian and international hobby-electronics suppliers. Prices are indicative 2026 retail figures in Indian rupees and will drift — treat them as a budgeting guide, not a quotation.

Estimated total: ₹2,600, excluding tools, shipping and consumables.

Tools and consumables

• Soldering iron (temperature controlled, 350 °C) with 0.8 mm 60/40 or lead-free solder
• Digital multimeter — continuity, DC volts and current ranges
• Wire strippers, flush cutters and a small set of precision screwdrivers
• Heat-shrink tubing and a heat gun (or a lighter, carefully)
• A laptop with a USB port and the toolchain listed above

Hardware Specifications

Consolidated electrical and interface specifications for every active part in the build.

Power Budget & Supply Sizing

Add up the typical active current of every part, then size the supply with at least 50 % headroom so transmit bursts and motor inrush never brown out the controller.

Summed typical draw is 250 mA. With a 1.5× design margin the supply should deliver at least 400 mA continuously at the stated rail voltage.

Software Requirements & Development Environment

Reference toolchain: Python 3.11 + PyTorch / OCR. Anything newer normally works; anything older may lack the board definitions used here.

• Install the Arduino IDE 2.3.x (or PlatformIO if you prefer a real editor and dependency locking).
• Add https://espressif.github.io/arduino-esp32/package_esp32_index.json under File → Preferences → Additional Board Manager URLs, then install esp32 from the Boards Manager.
• Set the correct port under Tools → Port. On Linux add yourself to the dialout group: sudo usermod -aG dialout $USER and log out and back in.
• Open the Serial Monitor at 115200 baud — every sketch here logs its state there.
• Keep File → Preferences → Show verbose output during: compilation switched on while you are debugging build errors.

Required libraries

Block Diagram

The block diagram shows the functional decomposition of the system — what senses, what decides, what acts, and where the data ends up.

Circuit Diagram & Wiring

The "wiring" is the OCR pipeline data flow — a document image is segmented into text regions, each recognised (character or sequence), then corrected by a language model into editable text.

Wire one row at a time and tick it off — most "it does not work" reports trace back to a single swapped pair.

Wiring explanation

• A scanner/camera provides the document image (quality strongly affects accuracy).
• Segment the page into lines/words/characters (or feed whole lines to a sequence model).
• Recognise digits/characters with a CNN, or sequences with a sequence recogniser.
• Post-process with a dictionary/language model to correct errors.
• Output editable, searchable text.

System Architecture

Read the stack from the bottom up: physical hardware, the firmware that drives it, the transport that moves data off the device, and the software a human actually looks at.

Working Principle

OCR is best understood as a ladder, and the bottom rung is the whole of classical deep-learning pedagogy: classify one character. The MNIST dataset — tens of thousands of 28×28 handwritten digits — is the "hello world" of neural networks precisely because it isolates that core problem cleanly. A small convolutional neural network learns to recognise the visual features of each digit (loops, strokes, junctions) and map an image to one of ten classes with very high accuracy. Building and training that classifier teaches every fundamental — convolutions, pooling, a softmax output, a loss, gradient descent — in a problem simple enough to master, which is why it is the universal starting point.

The leap from MNIST to real OCR is the leap from a pre-segmented single character to a page of connected text, and it introduces the two things MNIST hides. The first is segmentation/detection: a real document must be broken into lines, then words, then possibly characters, before anything can be recognised — and doing this reliably (handling spacing, skew, touching characters) is a substantial problem in itself. The second is that per-character cutting is brittle — characters touch, overlap, and vary in width — so modern OCR often skips it, feeding a whole line image to a sequence recogniser (a CNN+RNN with a CTC loss, or a sequence-to-sequence model) that reads the line as a sequence of characters without needing them pre-cut. This sequence approach is what makes robust line/word recognition possible.

The third rung is language-aware post-processing, and it is what separates a raw recogniser from a usable OCR engine. Visual recognition alone makes errors that are obvious in linguistic context: it might read "recognise" as "recogmse" or "0" for "O". A dictionary or language model corrects these by preferring valid, likely words and sequences — using the statistics of the language to fix what the pixels got wrong. This is why real OCR is a vision and language system: the image proposes, the language model disposes, and the combination is far more accurate than either alone.

The honesty this project needs centres on the gulf between a benchmark and a robust tool. Printed text is comparatively easy; handwriting is genuinely hard, because of the vast variation between writers (everyone's hand is different) and even within a writer (the same person's "a" varies), plus cursive connection, messy layouts, and idiosyncratic shapes — a digit classifier that aces MNIST is a long way from reading a doctor's scrawl. Accuracy also depends heavily on image quality: resolution, contrast, lighting, skew and noise all degrade recognition, so scanning/deskewing/binarising the input matters as much as the model. And a real engine must handle layout, mixed fonts, and languages. Framed as a ladder — master the MNIST CNN core, add segmentation and sequence recognition, then language post-processing — the project delivers a genuinely useful digitisation engine while teaching, rung by rung, how a real applied-vision system is composed from a simple classifier.

The maths behind it

Single-character classification (MNIST)

CNN(28×28 image) → probabilities over {0..9}
  digit = argmax(probs)

The core: learn visual features → class. The "hello world"
of deep learning.

Sequence recognition (no cutting)

Feed a whole LINE image to a sequence model:

  CNN features → RNN/transformer → char sequence
  trained with CTC (align without pre-segmenting chars)

Avoids brittle per-character cutting — robust to touching
/overlapping characters.

Language-model correction

raw = recogniser(image)          # pixels propose
text = argmax_w  P(w | raw) · P(w)  # language disposes
  (dictionary / n-gram / LM prior over valid words)

"recogmse" → "recognise". Vision + language > either alone.

Program Flowchart

The firmware is a single cooperative loop. Nothing blocks for long, so networking, sensing and the user interface all stay responsive.

Assembly Instructions

Build on a breadboard first and only commit to solder once the whole system has run for an hour without a fault.

1. Master the MNIST character core

   Train a CNN to classify MNIST digits (and EMNIST characters) — the fundamental skill the whole engine builds on.

2. Add segmentation and sequence recognition

   Segment pages into lines/words, and recognise lines with a sequence model (CNN+RNN+CTC) rather than brittle per-character cutting.

3. Add language post-processing

   Correct raw recognition with a dictionary/language model, and output editable text.

Step-by-Step Implementation Guide

Work through these in order. Each step ends in something you can observe, so a failure is always localised to the step you just finished.

1. Classify a character, then read a line

   Show the MNIST core (single-character classification) and the real-OCR line recogniser plus language correction.

   pythonrecognise.py Copy Download

   import torch, torch.nn.functional as F
   
   # --- Rung 1: the MNIST core (single-character classification) ---
   def classify_digit(img28, cnn):
       with torch.no_grad():
           probs = F.softmax(cnn(img28[None]), dim=1)[0]
       return int(probs.argmax()), float(probs.max())        # digit + confidence
   
   # --- Rung 2-3: real OCR line (sequence recognition + language model) ---
   def read_line(line_img, seq_model, lm):
       raw = seq_model.recognise(line_img)   # whole line, no per-char cutting
       return lm.correct(raw)                # dictionary/LM fixes visual errors

   def classify_digit(img28, cnn):The MNIST core — mapping one image to one of ten digits — is the fundamental classification skill everything else builds on.

   raw = seq_model.recognise(line_img) # whole line, no per-char cuttingReal OCR feeds a whole line to a sequence model, avoiding brittle character cutting that fails on touching/overlapping text.

   return lm.correct(raw) # dictionary/LM fixes visual errorsA language model corrects errors the pixels got wrong — vision proposes, language disposes.

2. Compose the page pipeline

   Preprocess (deskew/binarise), segment lines, recognise and correct each, and assemble editable text.

Complete Source Code

The listing below is complete and compiles as written — there are no elided sections. Read the annotations under each block before you upload it.

#!/usr/bin/env python3
"""
Handwriting & OCR Engine

A ladder: (1) a CNN classifies single characters (the MNIST core);
(2) segmentation + a sequence recogniser read lines without brittle
character cutting; (3) a language model corrects errors into editable
text. Handwriting is far harder than print; image quality matters a lot.
"""
import torch, torch.nn as nn, torch.nn.functional as F

class DigitCNN(nn.Module):                     # the MNIST core
    def __init__(self):
        super().__init__()
        self.c1 = nn.Conv2d(1, 32, 3); self.c2 = nn.Conv2d(32, 64, 3)
        self.fc1 = nn.Linear(64*5*5, 128); self.fc2 = nn.Linear(128, 10)
    def forward(self, x):
        x = F.max_pool2d(F.relu(self.c1(x)), 2)
        x = F.max_pool2d(F.relu(self.c2(x)), 2)
        x = x.flatten(1)
        return self.fc2(F.relu(self.fc1(x)))   # logits over 10 digits

class OCREngine:
    def __init__(self, seq_model, lm):
        self.seq = seq_model; self.lm = lm     # line recogniser + language model

    def read_line(self, line_img):
        raw = self.seq.recognise(line_img)     # sequence recognition (no cutting)
        return self.lm.correct(raw)            # language-model post-processing

    def read_page(self, image):
        clean = deskew(binarise(denoise(image)))   # quality preprocessing FIRST
        lines = segment_lines(clean)                # detect/segment text
        return "\n".join(self.read_line(l) for l in lines)   # editable text

if __name__ == "__main__":
    # Rung 1: train DigitCNN on MNIST (the classic core).
    # Rung 2-3: OCREngine composes sequence recognition + LM into page OCR.
    engine = OCREngine(SequenceRecogniser(), LanguageModel())
    print(engine.read_page(load_image("note.png")))
    # Print is easier; handwriting is hard; scan quality strongly matters.

Configuration & Calibration

Configuration steps

• Configure the character CNN (MNIST/EMNIST) and the sequence recogniser.
• Configure segmentation and preprocessing (deskew/binarise/denoise).
• Configure the dictionary/language model for post-processing.
• Configure output format (editable/searchable text).

Calibration procedure

An uncalibrated sensor produces confident, precise, wrong numbers. Do this once per physical unit and record the constants.

1. Character core

   Verify high MNIST/EMNIST accuracy before building up.

2. Sequence + LM

   Tune CTC decoding and language-model weighting on realistic lines.

3. Preprocessing

   Ensure deskew/binarise/denoise materially improve real-page accuracy.

Dataset, Model & Training

Dataset

MNIST for the digit-classification core; line/page datasets (e.g. IAM handwriting) for sequence OCR; printed-text corpora for print.

Handwriting variation and image quality in the data shape real-world accuracy.

Data preprocessing

• Deskew, binarise, normalise and denoise the document image.
• Segment lines/words (or feed whole lines to a sequence model).
• Normalise character/line size; augment (rotation, elastic distortion) for robustness.

Hyperparameters

Training process

• Train the CNN on MNIST/EMNIST for the character core; train the sequence recogniser on line data with CTC.
• Augment for handwriting variation; validate on held-out writers.
• Tune language-model post-processing on realistic text.

Evaluation, Metrics & Deployment

Digit accuracy for MNIST; character error rate (CER) and word error rate (WER) for text — with handwriting far harder than print.

Figures from the reference training run described above — reproduce them before trusting your own changes.

Inference example

import torch, torch.nn.functional as F

def classify_digit(img28, cnn):               # MNIST core
    with torch.no_grad():
        probs = F.softmax(cnn(img28[None]), dim=1)[0]
    return int(probs.argmax()), float(probs.max())

def read_line(line_img, seq_model, lm):       # real OCR: no char cutting
    raw = seq_model.recognise(line_img)       # CNN+RNN+CTC over the line
    text = lm.correct(raw)                     # dictionary/LM fixes errors
    return text                                # "recogmse" -> "recognise"

def ocr_page(image, segmenter, seq_model, lm):
    lines = segmenter.lines(deskew(binarise(image)))   # quality preprocessing
    return "\n".join(read_line(l, seq_model, lm) for l in lines)
    # Handwriting is far harder than print; image quality matters a lot.

Testing Procedure & Expected Output

Test from the bottom up. Confirm power, then each sensor in isolation, then the integrated loop — the first failing step tells you exactly where to look.

Bench-test checklist. If a row fails, stop and fix it before moving on.

Expected output

Editable text from images, from single digits to full pages, corrected by a language model.

{
  "mnist_digit": 7,
  "digit_confidence": 0.99,
  "line_raw": "the quick brown f0x",
  "line_corrected": "the quick brown fox",
  "note": "print easier than handwriting; quality matters"
}

The MNIST core classifies a digit with high confidence, and the line recogniser plus language model turns "brown f0x" into "brown fox" — the ladder from single character to corrected text.

Troubleshooting: Common Errors & Fixes

Performance Optimisation

• Master the character core before building the pipeline.
• Use sequence recognition to avoid brittle character cutting.
• Add language-model post-processing for real accuracy.
• Invest in image-quality preprocessing (deskew/binarise/denoise).
• Replace every delay() with a millis() comparison — blocking delays are the single most common cause of dropped readings.
• Sample sensors on a fixed cadence and publish on a slower one; you almost never need to transmit at the sampling rate.
• Move networking into its own FreeRTOS task so a slow DNS lookup cannot stall the control loop.
• Use uint8_t / uint16_t where the range allows; on an 8-bit AVR a 32-bit add costs four times as much.
• Profile before optimising — print micros() deltas around each stage and fix the slowest one first.

Safety Precautions

• OCR makes errors — do not use raw output for critical data without verification.
• Documents may contain personal/sensitive data — handle securely and lawfully.
• Handwriting accuracy is limited — set expectations honestly.
• Validate on representative documents before relying on the engine.
• Wear eye protection when soldering or cutting, and solder in a ventilated space — rosin flux fumes are a respiratory irritant.
• Power the circuit through a bench supply with a current limit while you are testing. A 300 mA limit turns a wiring mistake into a beep instead of a dead board.
• Disconnect power before changing any wiring. Hot-plugging a sensor onto a live bus is the fastest way to lose a controller.

Maintenance

• Retrain/extend for new fonts, scripts and handwriting.
• Improve preprocessing for new capture conditions.
• Update the language model/dictionary for the domain.
• Monitor CER/WER on real documents.
• Re-check every screw terminal and header after the first week — thermal cycling loosens connections that felt tight on day one.
• Recalibrate at the interval given in the calibration section, and keep the constants in a text file next to the firmware — not only in flash.
• Keep a short logbook of firmware versions and what changed. Six months later you will not remember why that constant is 1.083.

Future Improvements & Upgrades

A working v1 is a platform, not a finish line. These are the upgrades that add the most capability for the least rework.

• Add full page-layout analysis (columns, tables, forms).
• Add multilingual/script support.
• Add end-to-end transformer OCR.
• Add handwriting-specific writer adaptation.
• Design a proper PCB. Once the breadboard version has run for a month, moving to a two-layer board removes the intermittent-contact failures that dominate prototype faults.
• Add connectivity — an ESP32 and an MQTT publish turn a local gadget into something you can graph, alert on and analyse over months.
• Add persistent local storage (microSD or the on-chip flash) so a network outage does not create a hole in your data.
• Move configuration out of the source: a captive-portal setup page or a JSON config file makes the build reusable without a recompile.
• Add a battery and solar option so the unit survives a power cut and can be sited away from a socket.
• Write a small test harness that feeds synthetic sensor values through the decision logic, so you can validate thresholds without physically triggering the event.

Frequently Asked Questions

References & Learning Resources

These are the primary sources worth reading in full. Manufacturer datasheets always outrank forum posts when the two disagree.

1. Optical character recognitionReference
2. MNIST databaseReference
3. Connectionist temporal classification (CTC)Reference
4. Handwriting recognitionReference
5. IAM handwriting databaseDataset