Autonomous Lane-Keeping Car.

Project Overview

A vision-driven model car that steers itself to stay in its lane — the accessible core of self-driving, learned end-to-end or by detecting lanes.

Self-driving cars are dauntingly complex, but their most iconic sub-problem — staying in the lane — can be built at model-car scale and teaches the real ideas. This project builds an autonomous lane-keeping car: a small vehicle with a camera that steers itself to stay centred in its lane, using computer vision to see the road and a control loop to keep the car on it. It is the accessible, buildable core of autonomous driving — perception feeding control — on a platform you can hold.

There are two classic approaches, and both are worth understanding. The lane-detection approach is explicit: use computer vision to find the lane markings in the camera image (edge detection, colour thresholds, perspective transform to a bird's-eye view, fitting lane lines), compute how far the car is from the lane centre, and steer to correct — a transparent perception-then-control pipeline. The end-to-end learning approach is the famous alternative: train a neural network to map the camera image directly to a steering command, learning to drive by imitating a human driver's steering (behavioural cloning) — no explicit lane-finding, the network figures out what to look at. This project can do either, and comparing them is illuminating.

The value is the essential lesson of autonomy: closing the loop from perception to action in real time on a moving vehicle. It is honest that this is a controlled, simplified version of a hard problem — a model car on a clear track in good light is a world away from real roads, and lane-keeping is only one narrow slice of self-driving (no other cars, pedestrians, signs, intersections, weather, or edge cases). Real autonomy needs far more perception, redundancy and safety engineering, and end-to-end models are notoriously brittle outside their training distribution. But as a real-time, vision-driven lane-keeper — by explicit detection or end-to-end learning — it delivers a genuine, satisfying taste of autonomous driving and the clearest hands-on lesson in perception-to-control.

What this project does

• Steers a model car to stay centred in its lane
• Sees the road with a camera (computer vision)
• Detects lane markings, or learns steering end-to-end
• Computes lane offset and corrects steering
• Closes the perception-to-control loop in real time
• Demonstrates the core of autonomous driving
• Runs on edge hardware on a moving vehicle

Real-World Applications

Deployment contexts where a build of this kind earns its keep.

Features & Capabilities

• Vision-based lane keeping
• Lane-detection pipeline (edges/colour/bird's-eye/fit)
• End-to-end (behavioural cloning) option
• Real-time perception-to-control loop
• Steering control from lane offset
• Edge deployment on a model car
• Honest about the gap to real self-driving

Difficulty, Time & Required Skills

Skills you should have (or will pick up)

• Computer vision for lane detection
• Perspective transform and lane fitting
• End-to-end (behavioural cloning) driving
• Real-time perception-to-control loops
• Steering control and edge deployment

Bill of Materials

Every part below is commonly available from Indian and international hobby-electronics suppliers. Prices are indicative 2026 retail figures in Indian rupees and will drift — treat them as a budgeting guide, not a quotation.

Estimated total: ₹11,980, excluding tools, shipping and consumables.

Tools and consumables

• Soldering iron (temperature controlled, 350 °C) with 0.8 mm 60/40 or lead-free solder
• Digital multimeter — continuity, DC volts and current ranges
• Wire strippers, flush cutters and a small set of precision screwdrivers
• Heat-shrink tubing and a heat gun (or a lighter, carefully)
• A laptop with a USB port and the toolchain listed above

Hardware Specifications

Consolidated electrical and interface specifications for every active part in the build.

Power Budget & Supply Sizing

Add up the typical active current of every part, then size the supply with at least 50 % headroom so transmit bursts and motor inrush never brown out the controller.

Summed typical draw is 1686 mA. With a 1.5× design margin the supply should deliver at least 2600 mA continuously at the stated rail voltage.

Software Requirements & Development Environment

Reference toolchain: Python 3.11 + vision / deep learning. Anything newer normally works; anything older may lack the board definitions used here.

• Flash Raspberry Pi OS (64-bit) with Raspberry Pi Imager; pre-configure Wi-Fi, hostname and SSH in the Imager settings so the board comes up headless.
• Update first: sudo apt update && sudo apt full-upgrade -y, then reboot.
• Work inside a virtual environment — python3 -m venv ~/venv && source ~/venv/bin/activate. Bookworm blocks system-wide pip install by design.
• Enable the buses you need with sudo raspi-config → Interface Options (I²C, SPI, Serial, Camera).
• Develop over VS Code Remote-SSH so you edit on your laptop but run on the Pi.

Required libraries

Block Diagram

The block diagram shows the functional decomposition of the system — what senses, what decides, what acts, and where the data ends up.

Circuit Diagram & Wiring

The "wiring" combines a real vehicle and a perception-to-control loop — a forward camera feeds lane perception (detection or a learned network), which produces a steering command driving the car's motors.

Wire one row at a time and tick it off — most "it does not work" reports trace back to a single swapped pair.

Wiring explanation

• Mount a forward-facing camera to see the lane ahead.
• Run perception on the onboard edge computer (detection or a learned net).
• Compute a steering command (from lane offset, or directly from the network).
• Drive the car's steering/motors to stay centred.
• Use a controlled, well-lit marked track — this is a simplified slice of driving.

System Architecture

Read the stack from the bottom up: physical hardware, the firmware that drives it, the transport that moves data off the device, and the software a human actually looks at.

Working Principle

Autonomous driving is overwhelming as a whole, but it decomposes, and its cleanest, most iconic piece is lane keeping: keep the vehicle centred between the lane lines. Stripped to that, self-driving becomes a tractable, buildable problem — see the lane, decide how to steer, act on the steering, repeat — which is exactly the perception-to-control loop that sits at the heart of all robotics and autonomy. Building it on a model car teaches that loop for real, on a moving vehicle, without the crushing complexity of the full driving task.

The lane-detection approach makes perception explicit and is a superb computer-vision lesson. From the camera image you find the lane markings through a classic pipeline: isolate likely lane pixels (edge detection, colour/gradient thresholds), often apply a perspective transform to get a top-down "bird's-eye" view where lanes are parallel and easy to reason about, and fit lines or curves to the lane boundaries. From the fitted lanes you compute the car's offset from the lane centre (and the lane's curvature) — a concrete error signal. A controller (proportional/PID, exactly as in the line-follower) then turns that offset into a steering correction to drive the error to zero. Everything is transparent and debuggable: you can see what the car sees and why it steers.

The end-to-end learning approach is the famous alternative and a landmark idea in modern autonomy. Instead of hand-building the perception pipeline, you train a neural network to map the camera image directly to a steering command, learning by imitating a human driver — record images paired with the steering the human applied, and train the network to reproduce it (behavioural cloning). The network discovers on its own what visual features matter for steering, with no explicit lane-finding. It is remarkable that this works, and comparing the two approaches — transparent-but-hand-engineered detection versus powerful-but-opaque learning — is one of the most instructive parts of the project.

The honesty this project demands is about the vast gap between a lane-keeping model car and real self-driving, and it matters both technically and for safety framing. This is a controlled, simplified problem: a clear track in good lighting, no other vehicles, no pedestrians, no traffic signs or lights, no intersections, no weather, no darkness, no debris — none of the endless edge cases that make real autonomy staggeringly hard. Lane keeping is one narrow slice of driving; a real system needs vastly more perception (detecting and predicting every agent), sensor fusion and redundancy (cameras, radar, lidar), mapping, planning, and safety engineering built to fail safe. The end-to-end approach specifically is known to be brittle: it can steer beautifully on roads like its training data and fail unpredictably outside that distribution, and its opacity makes failures hard to diagnose — which is why real deployments do not simply clone a driver end-to-end. None of this diminishes the project; it frames it correctly. As a real-time, vision-driven lane-keeper — built by explicit detection, by end-to-end learning, or both — it delivers a genuine, hands-on taste of autonomy and the clearest possible lesson in closing the loop from perception to action, while being honest that it is the accessible doorway to self-driving, not the thing itself.

The maths behind it

Lane-detection pipeline

image → edges/colour threshold → bird's-eye (perspective)
      → fit lane lines/curves → lane CENTRE

offset = car_position − lane_centre   (the error signal)

Control (offset → steering)

steering = Kp·offset + Kd·d(offset)/dt   (PID, as in line-following)

Drive the offset to zero → stay centred. Real-time loop.

End-to-end (behavioural cloning)

train: network(camera image) → steering  (imitate human)
drive: steering = network(image)          # no explicit lane-finding

Powerful but OPAQUE and BRITTLE outside its training data.

Program Flowchart

The firmware is a single cooperative loop. Nothing blocks for long, so networking, sensing and the user interface all stay responsive.

Assembly Instructions

Build on a breadboard first and only commit to solder once the whole system has run for an hour without a fault.

1. Build the car and perception

   Mount a forward camera and edge computer on a model car, and implement lane perception (detection pipeline or an end-to-end network).

2. Close the perception-to-control loop

   Turn lane offset (or the network output) into a steering command and drive the car, in real time.

3. Tune and compare approaches

   Tune the controller / train the end-to-end model, and compare transparent detection with learned steering.

Step-by-Step Implementation Guide

Work through these in order. Each step ends in something you can observe, so a failure is always localised to the step you just finished.

1. Perceive the lane and steer

   Detect the lane offset (or run the end-to-end network) and produce a steering command in real time.

   pythondrive.py Copy Download

   import cv2
   
   def lane_offset(frame):
       edges = cv2.Canny(cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY), 50, 150)
       warp = birds_eye(edges)                     # top-down view (perspective)
       left, right = fit_lane_lines(warp)           # fit lane boundaries
       return (left + right) / 2 - frame.shape[1] / 2   # offset from centre
   
   def drive(frame, prev_offset, model=None, Kp=0.4, Kd=0.2):
       if model is not None:
           return float(model(preprocess(frame))), prev_offset   # end-to-end steering
       off = lane_offset(frame)                      # explicit detection
       steering = Kp * off + Kd * (off - prev_offset)   # PID -> steering
       return steering, off

   warp = birds_eye(edges) # top-down view (perspective)A perspective transform to a bird's-eye view makes lane lines parallel and easy to fit — a key step in the detection pipeline.

   return (left + right) / 2 - frame.shape[1] / 2 # offset from centreThe lane offset is the error signal — how far the car is from the lane centre — that the controller drives to zero.

   return float(model(preprocess(frame))), prev_offset # end-to-end steeringThe end-to-end route maps the image straight to a steering command, with no explicit lane-finding — powerful but opaque.

   steering = Kp * off + Kd * (off - prev_offset) # PID -> steeringPID control turns lane offset into a smooth steering correction, exactly as in a line-follower.

2. Run the loop in real time and compare

   Drive the loop fast enough to handle curves, and compare the transparent detection route with the learned end-to-end route.

Complete Source Code

The listing below is complete and compiles as written — there are no elided sections. Read the annotations under each block before you upload it.

#!/usr/bin/env python3
"""
Autonomous Lane-Keeping Car

Keeps a model car centred in its lane in real time via PERCEPTION →
CONTROL. Route A: explicit lane detection (edges/colour, bird's-eye,
fit) + PID. Route B: end-to-end CNN mapping image → steering
(behavioural cloning). A CONTROLLED, SIMPLIFIED slice of self-driving —
the accessible doorway, not real autonomy.
"""
import cv2

class LaneKeepingCar:
    def __init__(self, car, model=None, Kp=0.4, Kd=0.2):
        self.car = car; self.model = model      # model set = end-to-end route
        self.Kp, self.Kd = Kp, Kd; self.prev = 0.0

    def lane_offset(self, frame):               # Route A: explicit detection
        edges = cv2.Canny(cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY), 50, 150)
        warp = birds_eye(edges)                 # perspective transform
        left, right = fit_lane_lines(warp)      # fit lane boundaries
        return (left + right) / 2 - frame.shape[1] / 2   # offset from centre

    def steering(self, frame):
        if self.model is not None:              # Route B: end-to-end
            return float(self.model(preprocess(frame)))   # image -> steering
        off = self.lane_offset(frame)            # Route A: detect + PID
        s = self.Kp * off + self.Kd * (off - self.prev)
        self.prev = off
        return s

    def run(self):
        for frame in self.car.camera():          # real-time loop on the car
            self.car.steer(self.steering(frame)) # perception -> control
            self.car.drive(speed=SAFE_SPEED)

if __name__ == "__main__":
    LaneKeepingCar(ModelCar()).run()             # detection route (transparent)
    # Controlled track only; one narrow slice of driving; E2E is brittle/opaque.

Configuration & Calibration

Configuration steps

• Configure the camera, edge compute and car control.
• Configure the approach (detection pipeline vs end-to-end model).
• Configure detection thresholds/perspective or the trained network.
• Configure PID gains, loop rate and safe speed.

Calibration procedure

An uncalibrated sensor produces confident, precise, wrong numbers. Do this once per physical unit and record the constants.

1. Detection

   Tune edge/colour thresholds and the perspective transform for reliable lane finding on the track.

2. Control

   Tune PID for smooth centring; keep speed modest so the loop can react.

3. End-to-end

   Collect balanced data with recovery examples; validate on the track; expect off-distribution brittleness.

Dataset, Model & Training

Dataset

Lane-detection needs little/no training data (classic CV). End-to-end needs recorded image–steering pairs from human driving on the track.

End-to-end quality is bounded by how well the training runs cover the conditions.

Data preprocessing

• Detection: crop/ROI, colour/edge thresholds, perspective transform.
• End-to-end: crop, resize, normalise images; pair with steering; augment.
• Balance straight vs turning examples so the model does not just go straight.

Hyperparameters

Training process

• Detection: little training — tune CV and PID. End-to-end: collect image–steering data and train a CNN.
• Augment and include recovery examples so end-to-end can re-centre.
• Validate on the track; expect brittleness for end-to-end off-distribution.

Evaluation, Metrics & Deployment

Success is practical — staying centred through straights and curves in real time — with the honest caveat that this is a controlled, narrow slice of driving.

Figures from the reference training run described above — reproduce them before trusting your own changes.

Inference example

import cv2, numpy as np

# ---- Route A: explicit lane detection + PID control ----
def lane_offset(frame):
    edges = cv2.Canny(cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY), 50, 150)
    warp = birds_eye(edges)                    # perspective transform
    left, right = fit_lane_lines(warp)          # fit boundaries
    lane_center = (left + right) / 2
    return lane_center - frame.shape[1] / 2     # offset from car centre

def steer_pid(offset, prev, Kp=0.4, Kd=0.2):
    return Kp * offset + Kd * (offset - prev)   # offset -> steering

# ---- Route B: end-to-end (behavioural cloning) ----
def steer_end_to_end(frame, model):
    return float(model(preprocess(frame)))      # image -> steering directly
    # Opaque + brittle outside training data; this is a controlled slice of driving.

Testing Procedure & Expected Output

Test from the bottom up. Confirm power, then each sensor in isolation, then the integrated loop — the first failing step tells you exactly where to look.

Bench-test checklist. If a row fails, stop and fix it before moving on.

Expected output

A model car that keeps its lane in real time, by explicit detection or learned steering.

approach:  lane-detection + PID
lane_offset: -14 px (left of centre)
steering:   +5.9 (correct right)
loop:       28 FPS
state:      LANE-KEEPING (controlled track)

The car is left of centre and steering back, running the perception-to-control loop at real-time rate — a genuine taste of autonomy on a controlled track, not real-world self-driving.

Troubleshooting: Common Errors & Fixes

Performance Optimisation

• Close the perception-to-control loop fast enough for curves.
• Keep speed modest so the controller can react.
• For detection, use a bird's-eye view and robust thresholds.
• For end-to-end, include recovery data and augment.
• Pin the hot loop to one core with taskset and leave the others free for the OS.
• Prefer MJPEG over raw YUY2 when capturing from USB cameras — the decode cost is far lower than the USB bandwidth cost.
• Log to a tmpfs RAM disk and flush to the SD card once a minute; per-sample SD writes are what kills cards.
• Run the service under systemd with Restart=always so a crash never means a dead deployment.
• Profile before optimising — print micros() deltas around each stage and fix the slowest one first.

Safety Precautions

• This is a controlled, simplified slice of self-driving — do not treat it as real autonomy.
• End-to-end models are brittle and opaque off their training data — do not rely on them beyond the track.
• Keep speeds low and test in a safe, enclosed area.
• Real autonomy needs far more perception, redundancy and fail-safe engineering.
• Lithium cells vent and burn when abused. Only use protected cells or a proper BMS, never charge below 0 °C, and never leave a charging pack unattended on a wooden desk.
• Moving parts pinch. Keep fingers, cables and hair out of gear trains and wheels, and always test motion with the drivetrain unloaded and the robot on blocks first.
• Motors are inductive — always fit a flyback diode across a DC coil, or use a driver that already has one, or the back-EMF spike will destroy your GPIO.
• Wear eye protection when soldering or cutting, and solder in a ventilated space — rosin flux fumes are a respiratory irritant.
• Power the circuit through a bench supply with a current limit while you are testing. A 300 mA limit turns a wiring mistake into a beep instead of a dead board.
• Disconnect power before changing any wiring. Hot-plugging a sensor onto a live bus is the fastest way to lose a controller.

Maintenance

• Re-tune detection/PID for track and lighting changes.
• Retrain end-to-end for new conditions; keep recovery data.
• Check camera mounting and loop timing.
• Keep the scope framed honestly.
• Re-check every screw terminal and header after the first week — thermal cycling loosens connections that felt tight on day one.
• Log pack voltage. When resting voltage after a full charge drops below about 4.0 V, the cell is near end of life — replace it.
• Listen for gear chatter. A servo that buzzes at rest is either stripped or fighting a mechanical bind — fix it before it burns out.
• Recalibrate at the interval given in the calibration section, and keep the constants in a text file next to the firmware — not only in flash.
• Keep a short logbook of firmware versions and what changed. Six months later you will not remember why that constant is 1.083.

Future Improvements & Upgrades

A working v1 is a platform, not a finish line. These are the upgrades that add the most capability for the least rework.

• Add obstacle detection and stopping.
• Add sign/traffic-light recognition (toward more of driving).
• Add sensor fusion and mapping.
• Compare/deploy in a simulator for safety.
• Design a proper PCB. Once the breadboard version has run for a month, moving to a two-layer board removes the intermittent-contact failures that dominate prototype faults.
• Add connectivity — an ESP32 and an MQTT publish turn a local gadget into something you can graph, alert on and analyse over months.
• Add persistent local storage (microSD or the on-chip flash) so a network outage does not create a hole in your data.
• Move configuration out of the source: a captive-portal setup page or a JSON config file makes the build reusable without a recompile.
• Add a battery and solar option so the unit survives a power cut and can be sited away from a socket.
• Write a small test harness that feeds synthetic sensor values through the decision logic, so you can validate thresholds without physically triggering the event.

Frequently Asked Questions

References & Learning Resources

These are the primary sources worth reading in full. Manufacturer datasheets always outrank forum posts when the two disagree.

1. Self-driving carReference
2. Lane departure / keeping systemsReference
3. End-to-end learning for driving (behavioural cloning)Reference
4. Perspective transform (bird's-eye)Reference
5. PID controllerReference