Real-Time Object Detection.

Project Overview

A YOLO-based detector that finds and labels many objects in a live video stream at speed — the workhorse behind almost every vision application.

Almost every practical computer-vision application — surveillance, autonomous driving, retail analytics, robotics, quality inspection — rests on one capability: looking at an image and answering what objects are in it and where. That is object detection, and doing it in real time on a live video stream is the workhorse skill of applied vision. This project builds a real-time detector around the YOLO family (You Only Look Once), which reframed detection from a slow, multi-stage pipeline into a single fast network pass, making live detection on ordinary hardware possible.

The system takes frames from a camera or video, runs each through a trained detector, and outputs bounding boxes with class labels and confidence scores — "person 0.94 here, car 0.88 there" — drawn back onto the video live. The core ideas that make this work and make it fast are the single-pass architecture (the whole image is processed once, predicting all boxes together, rather than scanning region by region), non-maximum suppression (collapsing the many overlapping raw predictions into one clean box per object), and the speed/accuracy trade-off embodied in model size (a small model runs on a Pi at lower accuracy; a large one needs a GPU but detects more, smaller, harder objects).

The value is a reusable perception layer: once you can reliably detect and locate objects live, you can count them, track them, trigger on them, or feed them to downstream logic — which is why detection underpins so much. It is honest that a detector is only as good as its training data and classes (it detects what it was trained on, and struggles with unusual angles, small or occluded objects, and domain shift), that real-time performance depends heavily on hardware and model size, and that deployment (edge vs GPU) is a real engineering choice. But as a fast, accurate, live multi-object detector built on YOLO, it is both a genuinely useful building block and the single most important hands-on lesson in modern applied computer vision.

What this project does

• Detects and labels multiple objects in a live video stream
• Draws bounding boxes with class labels and confidence
• Runs in a single fast network pass (YOLO)
• Cleans overlapping predictions with non-maximum suppression
• Trades speed vs accuracy via model size (edge to GPU)
• Provides a reusable perception layer for counting/tracking/triggering
• Underpins surveillance, robotics, retail and autonomous vision

Real-World Applications

Deployment contexts where a build of this kind earns its keep.

Features & Capabilities

• Real-time multi-object detection (YOLO)
• Bounding boxes + labels + confidence
• Single-pass architecture (fast)
• Non-maximum suppression for clean boxes
• Model-size speed/accuracy trade-off
• Edge or GPU deployment
• Honest about training-data dependence and hard cases

Difficulty, Time & Required Skills

Skills you should have (or will pick up)

• Object detection concepts (boxes, classes, confidence)
• YOLO single-pass architecture and NMS
• Real-time inference on video (edge/GPU)
• Speed/accuracy trade-offs via model size
• Fine-tuning on custom classes / data

Bill of Materials

Every part below is commonly available from Indian and international hobby-electronics suppliers. Prices are indicative 2026 retail figures in Indian rupees and will drift — treat them as a budgeting guide, not a quotation.

Estimated total: ₹9,900, excluding tools, shipping and consumables.

Tools and consumables

• Soldering iron (temperature controlled, 350 °C) with 0.8 mm 60/40 or lead-free solder
• Digital multimeter — continuity, DC volts and current ranges
• Wire strippers, flush cutters and a small set of precision screwdrivers
• Heat-shrink tubing and a heat gun (or a lighter, carefully)
• A laptop with a USB port and the toolchain listed above

Hardware Specifications

Consolidated electrical and interface specifications for every active part in the build.

Power Budget & Supply Sizing

Add up the typical active current of every part, then size the supply with at least 50 % headroom so transmit bursts and motor inrush never brown out the controller.

Summed typical draw is 1450 mA. With a 1.5× design margin the supply should deliver at least 2200 mA continuously at the stated rail voltage.

Software Requirements & Development Environment

Reference toolchain: Python 3.11 + PyTorch / Ultralytics. Anything newer normally works; anything older may lack the board definitions used here.

• Flash Raspberry Pi OS (64-bit) with Raspberry Pi Imager; pre-configure Wi-Fi, hostname and SSH in the Imager settings so the board comes up headless.
• Update first: sudo apt update && sudo apt full-upgrade -y, then reboot.
• Work inside a virtual environment — python3 -m venv ~/venv && source ~/venv/bin/activate. Bookworm blocks system-wide pip install by design.
• Enable the buses you need with sudo raspi-config → Interface Options (I²C, SPI, Serial, Camera).
• Develop over VS Code Remote-SSH so you edit on your laptop but run on the Pi.

Required libraries

Block Diagram

The block diagram shows the functional decomposition of the system — what senses, what decides, what acts, and where the data ends up.

Circuit Diagram & Wiring

This is a software vision system; the "wiring" here is the data flow — a camera or video source feeds frames to the detector, which emits boxes, labels and confidences to the display and any downstream logic.

Wire one row at a time and tick it off — most "it does not work" reports trace back to a single swapped pair.

Wiring explanation

• A camera or video source provides frames to the detector.
• The detector runs a single forward pass producing raw boxes, labels and confidences.
• Non-maximum suppression collapses overlapping boxes into one per object.
• Clean detections are drawn on the video and/or passed to downstream logic.
• Choose model size for your hardware: small for edge, large for GPU accuracy.

System Architecture

Read the stack from the bottom up: physical hardware, the firmware that drives it, the transport that moves data off the device, and the software a human actually looks at.

Working Principle

Object detection is harder than classification: a classifier answers "what is this image?", but a detector must answer "what objects are present and where is each one?", which means localising a variable number of objects of different sizes anywhere in the frame. Early detectors did this slowly — proposing many candidate regions and classifying each — which was far too slow for video. YOLO's insight, and the reason it dominates real-time detection, is to treat detection as a single regression through one network: the whole image goes in once, and the network predicts all bounding boxes and their class probabilities together. "You Only Look Once" is literal — one pass, not thousands of region evaluations — and that is what makes live detection feasible.

Mechanically, the network divides the image into a grid and, at each location, predicts candidate boxes (position and size), an objectness/confidence score, and class probabilities. This produces many raw, overlapping predictions for the same object, so two clean-up steps follow. A confidence threshold discards weak predictions. Then non-maximum suppression (NMS) resolves the overlaps: among boxes that overlap heavily (high intersection-over-union) and predict the same class, it keeps the highest-confidence one and suppresses the rest, yielding a single clean box per object. Understanding NMS is essential, because without it a detector reports a messy pile of duplicate boxes.

The defining engineering dial is the speed/accuracy trade-off, set largely by model size. A small model (few parameters, low input resolution) runs fast — even on a Raspberry Pi or phone — but misses small, distant, or difficult objects and is less accurate. A large model detects more and harder objects accurately but needs a GPU to run at video rates. There is no universally right choice: an edge camera doing coarse person-detection wants the small model; a GPU server doing fine retail analytics wants the large one. Input resolution, batch size, and hardware (CPU vs GPU vs dedicated accelerator) all move the same dial. Choosing the point on this curve for the application and hardware is the core deployment decision.

The honest limits are as important as the capabilities. A detector only detects what it was trained on: a COCO-pretrained YOLO knows its ~80 everyday classes and nothing else, so detecting custom objects requires fine-tuning on labelled data — and the detector is only ever as good as that data's coverage of angles, lighting, scales, and occlusion. It struggles with small, distant, or occluded objects, unusual viewpoints, and domain shift (a model trained on daytime street scenes falters at night or indoors). Real-time performance is hardware-dependent and not guaranteed. And confidence scores are not calibrated probabilities to be trusted blindly. Within those bounds, though, a YOLO-based real-time detector is the single most reusable tool in applied vision — a fast, live "what and where" layer that almost every higher-level vision system is built upon, which is exactly why learning to build, run, and tune one is foundational.

The maths behind it

Detection output

For each detected object:
  box = (x, y, w, h)          # location + size
  class = argmax(class_probs)  # what it is
  confidence = objectness × class_prob

Keep only detections with confidence ≥ threshold.

Intersection over Union (IoU)

IoU(A,B) = area(A ∩ B) / area(A ∪ B)

Measures box overlap (0..1). Used by NMS to find duplicates
and by evaluation (a prediction matches truth if IoU ≥ 0.5).

Non-maximum suppression

sort detections by confidence (high → low)
repeat:
  keep the top box B
  remove any remaining box with IoU(B, ·) > NMS_thr
    AND same class
→ one clean box per object.

Speed / accuracy dial

small model / low-res  → fast, less accurate (edge)
large model / high-res → accurate, slower (GPU)

FPS ≈ compute / (model_cost × resolution)
Choose the point for your app + hardware.

Program Flowchart

The firmware is a single cooperative loop. Nothing blocks for long, so networking, sensing and the user interface all stay responsive.

Assembly Instructions

Build on a breadboard first and only commit to solder once the whole system has run for an hour without a fault.

1. Set up the detector and a video source

   Install the framework, load a pretrained YOLO model, and stream frames from a camera or video.

   Pick a model size that matches your hardware (small for edge, large for GPU).

2. Run inference with filtering and NMS

   Run each frame through the single-pass detector, filter by confidence, and apply NMS for one clean box per object.

3. Fine-tune and deploy

   Fine-tune on custom classes if needed, evaluate with mAP, and deploy at the right speed/accuracy point for the hardware.

Step-by-Step Implementation Guide

Work through these in order. Each step ends in something you can observe, so a failure is always localised to the step you just finished.

1. Detect, filter and clean per frame

   Run the detector on each frame, keep confident detections, and rely on NMS to remove duplicate overlapping boxes.

   pythonrealtime.py Copy Download

   from ultralytics import YOLO
   model = YOLO("yolov8s.pt")
   
   def detect(frame, conf=0.35, iou=0.45):
       # single forward pass; conf filter + NMS handled by the model
       r = model(frame, conf=conf, iou=iou)[0]
       out = []
       for b in r.boxes:
           out.append({
               "box": [int(v) for v in b.xyxy[0]],   # where
               "label": model.names[int(b.cls)],     # what
               "conf": float(b.conf),                # confidence
           })
       return out                                    # clean detections

   r = model(frame, conf=conf, iou=iou)[0]One forward pass detects all objects in the frame at once — the single-pass design that makes real time possible.

   "box": [int(v) for v in b.xyxy[0]], # whereEach detection carries its bounding box — the localisation that distinguishes detection from classification.

   "conf": float(b.conf), # confidenceConfidence lets downstream logic threshold detections; the conf/iou args apply the filter and NMS that yield one clean box per object.

2. Use the detections downstream

   Draw boxes/labels live, and feed detections to counting, tracking or triggering logic — the reusable perception layer.

Complete Source Code

The listing below is complete and compiles as written — there are no elided sections. Read the annotations under each block before you upload it.

#!/usr/bin/env python3
"""
Real-Time Object Detection (YOLO)

Detects and labels multiple objects live: single-pass inference,
confidence filtering + non-maximum suppression for clean boxes, and a
speed/accuracy dial via model size. A reusable perception layer for
counting, tracking and triggering. Detects only trained classes.
"""
from ultralytics import YOLO
import cv2, time

# Model size = the speed/accuracy dial: n/s for edge, l/x for GPU.
model = YOLO("yolov8s.pt")
CONF, NMS_IOU = 0.35, 0.45

def annotate(frame, dets):
    for d in dets:
        x1, y1, x2, y2 = d["box"]
        cv2.rectangle(frame, (x1, y1), (x2, y2), (0, 255, 0), 2)
        cv2.putText(frame, f'{d["label"]} {d["conf"]:.2f}', (x1, y1-6),
                    cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 1)
    return frame

def detect(frame):
    r = model(frame, conf=CONF, iou=NMS_IOU, verbose=False)[0]  # single pass + NMS
    return [{"box": [int(v) for v in b.xyxy[0]],
             "label": model.names[int(b.cls)],
             "conf": float(b.conf)} for b in r.boxes]

def main(source=0):
    cap = cv2.VideoCapture(source)
    while True:
        ok, frame = cap.read()
        if not ok: break
        t = time.time()
        dets = detect(frame)                      # what + where + confidence
        # ---- downstream hooks: count / track / trigger on 'dets' ----
        fps = 1.0 / max(time.time() - t, 1e-6)
        frame = annotate(frame, dets)
        cv2.putText(frame, f"{fps:.1f} FPS  {len(dets)} objects", (8, 24),
                    cv2.FONT_HERSHEY_SIMPLEX, 0.6, (0, 200, 255), 2)
        cv2.imshow("Real-Time Detection", frame)
        if cv2.waitKey(1) == 27: break
    cap.release(); cv2.destroyAllWindows()

if __name__ == "__main__":
    main(0)                                       # 0 = webcam; or a video path

Configuration & Calibration

Configuration steps

• Configure the model variant (speed/accuracy) for your hardware.
• Configure confidence and NMS-IoU thresholds.
• Configure the video source and (if custom) the fine-tuned classes.
• Configure downstream hooks (count/track/trigger).

Calibration procedure

An uncalibrated sensor produces confident, precise, wrong numbers. Do this once per physical unit and record the constants.

1. Thresholds

   Tune confidence to balance misses vs false detections; adjust NMS-IoU if boxes merge or duplicate.

2. Model size

   Pick the smallest model that meets your accuracy need at real-time FPS on the hardware.

3. Custom classes

   If fine-tuning, validate mAP on held-out images covering hard cases.

Dataset, Model & Training

Dataset

Detectors are trained on images annotated with bounding boxes and class labels. A COCO-pretrained model gives ~80 common classes out of the box; custom classes need your own labelled images.

Coverage of angles, scales, lighting and occlusion in the training data directly determines real-world accuracy.

Data preprocessing

• Resize/letterbox frames to the model input size; normalise pixels.
• Augment training data (scale, flip, mosaic, colour) for robustness.
• Ensure annotations are accurate — box quality caps model quality.

Model architecture

A YOLO detector has a backbone (feature extraction), a neck (multi-scale feature fusion), and a head predicting boxes/objectness/classes at several scales — enabling detection of both large and small objects in one pass.

Model variants (n/s/m/l/x) trade parameters and input size for speed vs accuracy.

Hyperparameters

Training process

• Start from pretrained weights and fine-tune on your labelled classes (transfer learning) — far less data than training from scratch.
• Augment heavily; validate on held-out images covering hard cases.
• Track mAP; watch for overfitting to a narrow set of conditions.

Evaluation, Metrics & Deployment

Detection quality is measured by mean Average Precision (mAP) across classes and IoU thresholds, alongside real-time speed (FPS) on the target hardware.

Figures from the reference training run described above — reproduce them before trusting your own changes.

Inference example

from ultralytics import YOLO
import cv2

model = YOLO("yolov8n.pt")          # small = edge-friendly; swap for l/x on GPU
CONF = 0.35

cap = cv2.VideoCapture(0)           # live camera
while True:
    ok, frame = cap.read()
    if not ok: break
    # single-pass detection + built-in NMS
    results = model(frame, conf=CONF)[0]
    for b in results.boxes:
        x1, y1, x2, y2 = map(int, b.xyxy[0])
        label = model.names[int(b.cls)]           # what
        conf  = float(b.conf)                     # confidence
        cv2.rectangle(frame, (x1, y1), (x2, y2), (0, 255, 0), 2)   # where
        cv2.putText(frame, f"{label} {conf:.2f}", (x1, y1 - 6),
                    cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 1)
    cv2.imshow("detections", frame)
    if cv2.waitKey(1) == 27: break

Testing Procedure & Expected Output

Test from the bottom up. Confirm power, then each sensor in isolation, then the integrated loop — the first failing step tells you exactly where to look.

Bench-test checklist. If a row fails, stop and fix it before moving on.

Expected output

A live video with clean labelled boxes and confidences, plus a detections stream for downstream use.

[
  { "label": "person", "conf": 0.94, "box": [220, 90, 310, 360] },
  { "label": "car",    "conf": 0.88, "box": [400, 210, 560, 300] },
  { "label": "dog",    "conf": 0.81, "box": [120, 260, 210, 350] }
]

Three objects detected in one frame with locations and confidences — the "what and where" layer other vision logic builds on.

Troubleshooting: Common Errors & Fixes

Performance Optimisation

• Pick the smallest model meeting accuracy at real-time FPS.
• Tune confidence and NMS-IoU for clean, correct boxes.
• Use a GPU/accelerator or lower resolution for speed.
• Fine-tune on representative data for the real conditions.
• Pin the hot loop to one core with taskset and leave the others free for the OS.
• Prefer MJPEG over raw YUY2 when capturing from USB cameras — the decode cost is far lower than the USB bandwidth cost.
• Log to a tmpfs RAM disk and flush to the SD card once a minute; per-sample SD writes are what kills cards.
• Run the service under systemd with Restart=always so a crash never means a dead deployment.
• Profile before optimising — print micros() deltas around each stage and fix the slowest one first.

Safety Precautions

• A detector is not infallible — do not rely on it alone for safety-critical decisions without redundancy and validation.
• Cameras raise privacy obligations — follow notice/consent rules and minimise stored imagery.
• Confidence scores are not guarantees; validate mAP before trusting it.
• Beware bias from unrepresentative training data.
• Wear eye protection when soldering or cutting, and solder in a ventilated space — rosin flux fumes are a respiratory irritant.
• Power the circuit through a bench supply with a current limit while you are testing. A 300 mA limit turns a wiring mistake into a beep instead of a dead board.
• Disconnect power before changing any wiring. Hot-plugging a sensor onto a live bus is the fastest way to lose a controller.

Maintenance

• Retrain/fine-tune as conditions or classes change.
• Monitor accuracy for domain drift over time.
• Keep thresholds tuned to the deployment.
• Track FPS as models/hardware change.
• Re-check every screw terminal and header after the first week — thermal cycling loosens connections that felt tight on day one.
• Recalibrate at the interval given in the calibration section, and keep the constants in a text file next to the firmware — not only in flash.
• Keep a short logbook of firmware versions and what changed. Six months later you will not remember why that constant is 1.083.

Future Improvements & Upgrades

A working v1 is a platform, not a finish line. These are the upgrades that add the most capability for the least rework.

• Add multi-object tracking (assign persistent IDs across frames).
• Add segmentation (pixel masks) or pose on top of detection.
• Quantise/prune for faster edge inference.
• Add active learning to target hard cases in retraining.
• Design a proper PCB. Once the breadboard version has run for a month, moving to a two-layer board removes the intermittent-contact failures that dominate prototype faults.
• Add connectivity — an ESP32 and an MQTT publish turn a local gadget into something you can graph, alert on and analyse over months.
• Add persistent local storage (microSD or the on-chip flash) so a network outage does not create a hole in your data.
• Move configuration out of the source: a captive-portal setup page or a JSON config file makes the build reusable without a recompile.
• Add a battery and solar option so the unit survives a power cut and can be sited away from a socket.
• Write a small test harness that feeds synthetic sensor values through the decision logic, so you can validate thresholds without physically triggering the event.

Frequently Asked Questions

References & Learning Resources

These are the primary sources worth reading in full. Manufacturer datasheets always outrank forum posts when the two disagree.

1. Object detectionReference
2. YOLO (You Only Look Once)Reference
3. Non-maximum suppression / IoUReference
4. Ultralytics YOLODocs
5. COCO datasetDataset