Contents — 27 sections
Project Overview
A field camera node that photographs crop leaves, runs a trained disease classifier on-device, and flags early signs of infection — with a clear account of what the model can and cannot recognise.
Plant disease, caught early, is often manageable — a targeted fungicide, removing affected plants, adjusting conditions. Caught late, it can take a field. But early detection requires close, frequent inspection of leaves that a farmer covering acres cannot do, and the visual signs are subtle and easy to miss until they are not. A camera node that automatically photographs leaves and runs a disease classifier on the images offers a way to inspect continuously and flag problems while they are still small.
This is a genuine and active application of edge machine learning, and it is also one where honesty about the limits is essential. A model trained to recognise a specific set of diseases on a specific set of crops does that well; shown a disease it was never trained on, or a crop it never saw, or a nutrient deficiency that looks like a disease, it will confidently produce a wrong answer. The model recognises patterns it has learned, not "disease" in the abstract. This documentation makes the scope of the model — which crops, which diseases — central, and frames the output as a flag for confirmation, not a diagnosis.
The build runs a trained convolutional network on an ESP32-S3 (with its vector instructions and PSRAM for on-device inference) or, for larger models, streams to a Raspberry Pi. It captures a leaf image, runs the classifier, and reports the most likely class with its confidence — flagging low-confidence or disease-positive results for a human to confirm. The famous PlantVillage dataset (54,000 labelled leaf images across many crops and diseases) is the standard training set, and its strengths and well-known weaknesses (lab conditions, single leaves on plain backgrounds) are discussed honestly, because a model trained on it can perform far worse in a real field.
The result is a node that continuously watches a crop and surfaces likely disease early, understood correctly as a screening tool whose job is to say "look at this, it might be diseased" to a person who can then confirm and act.
What this project does
- Captures leaf images automatically on a schedule or trigger.
- Runs a trained crop-disease classifier on-device (ESP32-S3) or on a Raspberry Pi.
- Reports the most likely disease class and confidence for a defined set of crops and diseases.
- Flags disease-positive and low-confidence results for human confirmation.
- Handles the model's scope honestly — reports "unknown" outside its training.
- Logs detections with images for review and for retraining.
- Reports over the network for field-scale coverage.
Real-World Applications
| Setting | How it is used |
|---|---|
| Early disease detection | Flagging likely infection while it is still small and manageable. |
| Greenhouse crop monitoring | Controlled conditions where the model performs closest to its training. |
| High-value crops | Where the cost of a missed disease justifies dense camera coverage. |
| Scouting assistance | Directing a scout's attention to plants the model flags. |
| Learning edge AI | A complete train-quantise-deploy pipeline for a real vision task. |
| Research and demonstration | A platform for field-vision disease-detection studies. |
Deployment contexts where a build of this kind earns its keep.
Features & Capabilities
- On-device inference with a quantised CNN on the ESP32-S3, or a larger model on a Pi.
- Defined, honest scope — specific crops and diseases, "unknown" outside them.
- Confidence-gated flagging so uncertain results go to a human, not acted on automatically.
- PlantVillage-trained baseline with the real-field performance gap discussed openly.
- Field-image robustness techniques (augmentation, background handling) to narrow that gap.
- Detection logging with images for confirmation and for collecting field data to retrain.
- Scheduled or triggered capture for continuous or event-based inspection.
- Network reporting for field-scale coverage.
Difficulty, Time & Required Skills
| Attribute | Value |
|---|---|
| Difficulty level | Advanced |
| Estimated completion time | 20–30 hours |
| Indicative build cost | ₹3,200 – ₹6,500 depending on compute choice |
| Primary discipline | Agriculture |
| Reference platform | ESP32-S3 DevKitC-1 |
Skills you should have (or will pick up)
- Python and deep learning (CNN training, transfer learning)
- Model quantisation and edge deployment (TFLite Micro / ONNX)
- Camera interfacing and image capture
- Understanding of dataset bias and real-world generalisation
- Honest evaluation and scope definition
Bill of Materials
Every part below is commonly available from Indian and international hobby-electronics suppliers. Prices are indicative 2026 retail figures in Indian rupees and will drift — treat them as a budgeting guide, not a quotation.
| Component | Key specification | Qty | Approx. cost |
|---|---|---|---|
| ESP32-S3 DevKitC-1 The vector extensions roughly triple TinyML inference speed over the original ESP32. | Dual-core Xtensa LX7 @ 240 MHz, 512 KB SRAM + 8 MB PSRAM, vector instructions for ML, Wi-Fi + BLE 5 | 1 | ₹900 |
| ESP32-CAM (AI-Thinker) + OV2640 No USB-serial on board — needs an FTDI/CP2102 adapter to flash. | ESP32-S chip, 4 MB PSRAM, 2 MP OV2640 camera, microSD slot, on-board LED flash | 1 | ₹620 |
| Raspberry Pi Zero 2 W 512 MB RAM — enable 1 GB zram swap before running any vision model. | Quad-core Cortex-A53 @ 1 GHz, 512 MB RAM, Wi-Fi 802.11n, BLE 4.2, CSI camera port | 1 | ₹2,200 |
| Raspberry Pi Camera Module 3 Pi 5 uses a narrower 22-pin CSI cable — the old 15-pin ribbon will not fit. | 12 MP IMX708, autofocus, HDR, 1080p50, CSI-2 ribbon | 1 | ₹2,600 |
| 5 V 3 A regulated SMPS adapter Measure the real output — many "3 A" adapters sag below 4.7 V at 2 A. | 100–240 VAC in, 5 V ±5 % out, 3 A, short-circuit and over-voltage protection | 1 | ₹350 |
| Double-sided perfboard 7 × 9 cm + headers Solder female headers so the MCU can be swapped without desoldering. | FR-4, 0.1″ pitch, plated through-holes, 24 × 18 grid | 1 | ₹60 |
| IP65 ABS junction enclosure 158 × 90 × 60 mm Fit cable glands, not drilled holes, or the IP rating means nothing. | IP65, ABS, −20 to +80 °C, transparent lid, wall-mount lugs | 1 | ₹260 |
| Camera + close-focus lens A leaf-disease image needs the leaf sharp and filling the frame — close focus and good light matter. | For sharp leaf-scale images at the working distance | 1 | ₹400 |
| Diffuse LED illumination Consistent, diffuse lighting reduces the appearance variation the model has to cope with. | Even white light for consistent images | 1 | ₹250 |
| Weatherproof camera housing | IP65, clear window, adjustable mount | 1 | ₹380 |
Estimated total: ₹8,020, excluding tools, shipping and consumables.
Tools and consumables
- Soldering iron (temperature controlled, 350 °C) with 0.8 mm 60/40 or lead-free solder
- Digital multimeter — continuity, DC volts and current ranges
- Wire strippers, flush cutters and a small set of precision screwdrivers
- Heat-shrink tubing and a heat gun (or a lighter, carefully)
- A laptop with a USB port and the toolchain listed above
Hardware Specifications
| Part | Specification | Supply | Interface | Reference |
|---|---|---|---|---|
| ESP32-S3 DevKitC-1 | Dual-core Xtensa LX7 @ 240 MHz, 512 KB SRAM + 8 MB PSRAM, vector instructions for ML, Wi-Fi + BLE 5 | 3.3 V logic / 5 V USB | USB-OTG, SPI, I²C, I²S, LCD/camera bus | Datasheet |
| ESP32-CAM (AI-Thinker) + OV2640 | ESP32-S chip, 4 MB PSRAM, 2 MP OV2640 camera, microSD slot, on-board LED flash | 5 V in / 3.3 V logic | UART (programmer required), SPI, I²C | Datasheet |
| Raspberry Pi Zero 2 W | Quad-core Cortex-A53 @ 1 GHz, 512 MB RAM, Wi-Fi 802.11n, BLE 4.2, CSI camera port | 5 V / 2 A micro-USB | GPIO, SPI, I²C, UART, CSI | Datasheet |
| Raspberry Pi Camera Module 3 | 12 MP IMX708, autofocus, HDR, 1080p50, CSI-2 ribbon | 3.3 V via CSI | CSI-2 | Datasheet |
| 5 V 3 A regulated SMPS adapter | 100–240 VAC in, 5 V ±5 % out, 3 A, short-circuit and over-voltage protection | 5 V | DC barrel / USB | Datasheet |
| Double-sided perfboard 7 × 9 cm + headers | FR-4, 0.1″ pitch, plated through-holes, 24 × 18 grid | — | — | Datasheet |
| IP65 ABS junction enclosure 158 × 90 × 60 mm | IP65, ABS, −20 to +80 °C, transparent lid, wall-mount lugs | — | — | Datasheet |
Consolidated electrical and interface specifications for every active part in the build.
Power Budget & Supply Sizing
Add up the typical active current of every part, then size the supply with at least 50 % headroom so transmit bursts and motor inrush never brown out the controller.
| Load | Supply rail | Typical current (mA) | Notes |
|---|---|---|---|
| ESP32-S3 DevKitC-1 | 3.3 V logic / 5 V USB | 180 | The vector extensions roughly triple TinyML inference speed over the original ESP32. |
| ESP32-CAM (AI-Thinker) + OV2640 | 5 V in / 3.3 V logic | 220 | No USB-serial on board — needs an FTDI/CP2102 adapter to flash. |
| Raspberry Pi Zero 2 W | 5 V / 2 A micro-USB | 350 | 512 MB RAM — enable 1 GB zram swap before running any vision model. |
| Raspberry Pi Camera Module 3 | 3.3 V via CSI | 250 | Pi 5 uses a narrower 22-pin CSI cable — the old 15-pin ribbon will not fit. |
| 5 V 3 A regulated SMPS adapter | 5 V | 3000 | Measure the real output — many "3 A" adapters sag below 4.7 V at 2 A. |
Summed typical draw is 4000 mA. With a 1.5× design margin the supply should deliver at least 6000 mA continuously at the stated rail voltage.
Software Requirements & Development Environment
Reference toolchain: Python 3.11 + TensorFlow/Keras for training; ESP32-S3 (Arduino/ESP-IDF) or Raspberry Pi OS for deployment. Anything newer normally works; anything older may lack the board definitions used here.
- Install the Arduino IDE 2.3.x (or PlatformIO if you prefer a real editor and dependency locking).
- Add
https://espressif.github.io/arduino-esp32/package_esp32_index.jsonunder File → Preferences → Additional Board Manager URLs, then install esp32 from the Boards Manager. - Set the correct port under Tools → Port. On Linux add yourself to the
dialoutgroup:sudo usermod -aG dialout $USERand log out and back in. - Open the Serial Monitor at 115200 baud — every sketch here logs its state there.
- Keep File → Preferences → Show verbose output during: compilation switched on while you are debugging build errors.
Required libraries
| Library | Why it is needed | Install |
|---|---|---|
| TensorFlow Lite for Microcontrollers 2.4.0-alpha | Int8 neural-network inference inside 200 KB of RAM. | Library Manager → "TensorFlowLite_ESP32" |
| Edge Impulse Arduino SDK per-project export | Deployable C++ bundle of a trained TinyML classifier. | Sketch → Include Library → Add .ZIP from the Edge Impulse export |
| Python 3.11+ | Runtime for the analysis, training and service code. | sudo apt install python3 python3-venv python3-pip |
| TensorFlow / Keras 2.17+ | High-level model building and the TFLite converter. | pip install tensorflow |
| NumPy 1.26+ | Vectorised array maths underpinning every other library here. | pip install numpy |
| OpenCV 4.10+ | Frame capture, colour conversion, drawing and classical CV operators. | pip install opencv-python |
| Ultralytics YOLO 8.3+ | Training and inference API for YOLOv8/v11 detectors. | pip install ultralytics |
| ONNX Runtime 1.19+ | Portable, quantised inference across CPU, GPU and NPUs. | pip install onnxruntime |
| Picamera2 0.3.20+ | libcamera-based capture API for Pi Camera modules. | sudo apt install python3-picamera2 |
Block Diagram
The block diagram shows the functional decomposition of the system — what senses, what decides, what acts, and where the data ends up.
Circuit Diagram & Wiring
Every signal line in the build is shown below, followed by a pin-by-pin connection table you can work through with a multimeter in hand.
| Peripheral | Peripheral pin | Controller pin | Signal |
|---|---|---|---|
| Camera (OV2640 / Pi CSI) | camera bus | on-board | Image capture |
| Illumination control | MOSFET | GPIO 4 | LED light during capture |
| Trigger (PIR/schedule) | OUT | GPIO 13 | Capture trigger |
| Status LED | Anode | GPIO 2 | Inference/result indicator |
| Wi-Fi/LoRa reporting | — | radio | Detection uplink |
Wire one row at a time and tick it off — most "it does not work" reports trace back to a single swapped pair.
Wiring explanation
- The camera and lens must produce a sharp, well-lit, leaf-scale image. Disease signs are fine — spots, lesions, discolouration — so a blurry or poorly-lit image defeats the model regardless of how good it is. Set the focus at the working distance and provide even illumination.
- Diffuse LED lighting reduces the lighting variation the model must handle. Harsh directional light creates shadows and specular highlights that look nothing like the training images, degrading accuracy.
- On the ESP32-S3, enable PSRAM and use the camera driver as in the doorbell project; the model runs in the vector-accelerated inference path.
- For larger, more accurate models, use a Raspberry Pi with the camera module — it can run a full-size CNN that the ESP32 cannot, at higher power and cost.
- Weatherproof the housing with a clear, clean window in front of the lens; a dirty or fogged window ruins the image. Position it so it images representative leaves.
- For field-scale coverage, either many cheap nodes at fixed positions or a smaller number on a moving platform (e.g. a rail or a robot) — the model and pipeline are the same.
System Architecture
Read the stack from the bottom up: physical hardware, the firmware that drives it, the transport that moves data off the device, and the software a human actually looks at.
Working Principle
The recognition is done by a convolutional neural network trained to classify leaf images into disease categories. A CNN learns, through training on labelled examples, to detect the visual features that distinguish classes — for disease, that means the spots, lesions, discolourations, mould and wilting patterns characteristic of each disease on each crop. Given enough labelled examples, it learns these patterns far better than hand-coded rules could, which is why deep learning dominates plant-disease vision.
The critical thing to understand is what the model actually learns, because it determines how the output must be used. The model learns to distinguish the classes it was trained on, based on the images it was shown. It does not learn "disease" as a concept — it learns "images that look like the tomato-early-blight examples versus images that look like the healthy-tomato examples." Consequently: shown a disease not in its training set, it will assign the nearest class it knows, confidently and wrongly. Shown a crop it never saw, its output is meaningless. Shown a nutrient deficiency or pest damage that visually resembles a trained disease, it may report that disease. The model is a pattern-matcher over a fixed set of classes, and its scope is exactly the crops and diseases it was trained on — no more.
This is why the dataset matters so much, and why the famous PlantVillage dataset needs honest discussion. PlantVillage contains about 54,000 images across 14 crops and 26 diseases, and it is the standard benchmark — models trained on it report accuracies above 99 %. But those images are single leaves, detached, photographed on a plain uniform background in controlled lighting. A model trained on them learns features that include the plain background and the controlled lighting. Deployed on a real field image — a leaf among other leaves, with soil and sky in the background, in dappled sunlight — the model often drops from 99 % to well below 50 %, because the real image looks nothing like its training. This "lab-to-field gap" is one of the most documented failures in agricultural computer vision, and pretending it does not exist is how these systems disappoint.
Narrowing the gap requires real field data and careful training. Fine-tuning on field images (even a modest number) helps enormously. Strong augmentation — varying background, lighting, scale, angle — during training makes the model more robust. And crucially, adding an "unknown" or low-confidence handling: rather than forcing every image into a known class, the model should be able to say "I do not recognise this confidently," which routes the image to a human rather than producing a confident wrong answer.
Deployment is a quantise-and-run edge pipeline. The model is trained in float, then quantised to int8 for the ESP32-S3 (which has vector instructions for exactly this) using TFLite Micro, or run at higher precision on a Raspberry Pi for a larger model. The trade-off is capability versus cost and power: the ESP32-S3 runs a small model cheaply and on little power (good for dense, distributed nodes); the Pi runs a larger, more accurate model at more cost and power (good for a smaller number of better nodes). Both are legitimate, and the choice depends on whether you want many modest eyes or fewer sharp ones.
The output is used as a flag, not a verdict. A disease-positive or low-confidence result is logged with its image and surfaced to a human, who confirms — the person is the diagnostic authority, the model is the attention-director that says "this leaf looks like it might have early blight, take a look." Used this way, across a crop no person can inspect leaf by leaf, it genuinely catches disease early. Used as an autonomous diagnostician, it would misfire on everything outside its narrow training and cause more harm than good.
The maths behind it
CNN classification and confidence
Input: leaf image, resized to model input (e.g. 224×224×3),
normalised.
CNN → logits → softmax → class probabilities p_c
predicted class = argmax_c p_c
confidence = max_c p_c
Decision:
if confidence < THRESHOLD: mark "uncertain" → human
else if class is a disease: flag for confirmation
else: healthy
A calibrated confidence is essential — an over-confident
model defeats the confidence gate.
Depthwise-separable model cost (edge)
MobileNet-style block (used for edge deployment):
standard conv: K²·C_in·C_out MACs/pixel
depthwise-sep: K²·C_in + C_in·C_out MACs/pixel
For K=3, C=128: ~8–9× cheaper — the reason mobile/edge
vision uses this decomposition.
Int8 quantisation: 4× smaller, several× faster on the
ESP32-S3 vector unit, typically <2% accuracy loss with
a representative calibration set.
The lab-to-field gap (why it matters)
PlantVillage (lab) test accuracy: >99%
Same model on real field images: often <50%
Causes: plain vs complex background,
controlled vs variable lighting, single vs
multiple leaves, scale and angle differences.
Mitigations (measured improvement):
strong background/lighting augmentation: +10–20%
fine-tune on field images: +20–40%
"unknown" class / confidence gating: fewer
confident errors
Report field accuracy, not lab accuracy.
Program Flowchart
The firmware is a single cooperative loop. Nothing blocks for long, so networking, sensing and the user interface all stay responsive.
Assembly Instructions
Build on a breadboard first and only commit to solder once the whole system has run for an hour without a fault.
Step-by-Step Implementation Guide
Work through these in order. Each step ends in something you can observe, so a failure is always localised to the step you just finished.
Complete Source Code
The listing below is complete and compiles as written — there are no elided sections. Read the annotations under each block before you upload it.
#!/usr/bin/env python3
"""Train a crop-disease classifier and export int8 TFLite for the edge.
Uses transfer learning from MobileNetV2. Emphasises the field-robustness
techniques (augmentation) and an explicit low-confidence path — because a
model that cannot say "I don't know" produces confident wrong answers on
anything outside its training.
"""
from __future__ import annotations
import pathlib
import numpy as np
import tensorflow as tf
IMG = 224
# Define the SCOPE explicitly: these crops and diseases and nothing else.
CLASSES = [
"tomato_healthy", "tomato_early_blight", "tomato_late_blight",
"tomato_leaf_mold", "tomato_septoria",
"potato_healthy", "potato_early_blight", "potato_late_blight",
# ... your defined scope. Off-scope images are NOT reliably classified.
]
def make_dataset(root: str, subset: str):
return tf.keras.utils.image_dataset_from_directory(
root, validation_split=0.2, subset=subset, seed=1,
image_size=(IMG, IMG), batch_size=32, class_names=CLASSES)
def build_model() -> tf.keras.Model:
base = tf.keras.applications.MobileNetV2(
input_shape=(IMG, IMG, 3), include_top=False, weights="imagenet")
base.trainable = False # transfer learning
# Strong augmentation is the single most effective lab-to-field mitigation.
augment = tf.keras.Sequential([
tf.keras.layers.RandomFlip("horizontal"),
tf.keras.layers.RandomRotation(0.2),
tf.keras.layers.RandomZoom(0.2),
tf.keras.layers.RandomBrightness(0.3),
tf.keras.layers.RandomContrast(0.3),
# Random background/crop simulates field clutter the lab data lacks.
tf.keras.layers.RandomTranslation(0.15, 0.15),
])
inp = tf.keras.Input((IMG, IMG, 3))
x = augment(inp)
x = tf.keras.applications.mobilenet_v2.preprocess_input(x)
x = base(x, training=False)
x = tf.keras.layers.GlobalAveragePooling2D()(x)
x = tf.keras.layers.Dropout(0.3)(x)
out = tf.keras.layers.Dense(len(CLASSES), activation="softmax")(x)
return tf.keras.Model(inp, out)
def main() -> None:
train = make_dataset("dataset", "training")
val = make_dataset("dataset", "validation")
model = build_model()
model.compile(optimizer="adam", loss="sparse_categorical_crossentropy",
metrics=["accuracy"])
model.fit(train, validation_data=val, epochs=20)
# Fine-tune the top of the base on FIELD images if you have them —
# this is what closes most of the lab-to-field gap.
model.get_layer("mobilenetv2_1.00_224").trainable = True
model.compile(optimizer=tf.keras.optimizers.Adam(1e-5),
loss="sparse_categorical_crossentropy", metrics=["accuracy"])
model.fit(train, validation_data=val, epochs=10)
# --- int8 quantisation for the ESP32-S3 ---
def rep():
for imgs, _ in train.take(50):
for i in range(imgs.shape[0]):
yield [imgs[i:i+1].numpy().astype(np.float32)]
conv = tf.lite.TFLiteConverter.from_keras_model(model)
conv.optimizations = [tf.lite.Optimize.DEFAULT]
conv.representative_dataset = rep
conv.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
conv.inference_input_type = tf.int8
conv.inference_output_type = tf.int8
tflite = conv.convert()
pathlib.Path("disease_int8.tflite").write_bytes(tflite)
print(f"model: {len(tflite)/1024:.0f} KB, {len(CLASSES)} classes")
print("SCOPE:", ", ".join(CLASSES))
print("Off-scope inputs are NOT reliably classified — gate on confidence.")
if __name__ == "__main__":
main()
/* ═══════════════════════════════════════════════════════════════
Crop Disease Camera — ESP32-S3 edge inference
Captures a leaf image, runs the quantised classifier, and flags
disease-positive or low-confidence results for a human to confirm.
The model recognises ONLY its trained crops and diseases. It flags
for confirmation; it does not diagnose.
══════════════════════════════════════════════════════════════════ */
#include <esp_camera.h>
#include <TensorFlowLite_ESP32.h>
#include "tensorflow/lite/micro/micro_interpreter.h"
#include "tensorflow/lite/micro/micro_mutable_op_resolver.h"
#include "model_data.h" // exported disease_int8.tflite
#define IMG 224
#define N_CLASSES 8
#define CONF_THRESHOLD 0.70f // below this → uncertain → human
const char *CLASSES[N_CLASSES] = {
"tomato_healthy", "tomato_early_blight", "tomato_late_blight",
"tomato_leaf_mold", "tomato_septoria",
"potato_healthy", "potato_early_blight", "potato_late_blight"
};
const bool IS_DISEASE[N_CLASSES] = { false, true, true, true, true, false, true, true };
static uint8_t *arena;
static tflite::MicroInterpreter *interp;
static TfLiteTensor *input, *output;
void inferenceBegin() {
arena = (uint8_t *)heap_caps_malloc(600 * 1024, MALLOC_CAP_SPIRAM);
static tflite::MicroMutableOpResolver<12> resolver;
resolver.AddConv2D(); resolver.AddDepthwiseConv2D();
resolver.AddRelu6(); resolver.AddAdd();
resolver.AddAveragePool2D(); resolver.AddReshape();
resolver.AddFullyConnected(); resolver.AddSoftmax();
resolver.AddPad(); resolver.AddMean();
resolver.AddQuantize(); resolver.AddDequantize();
static tflite::MicroInterpreter s(tflite::GetModel(g_model), resolver,
arena, 600 * 1024);
interp = &s;
interp->AllocateTensors();
input = interp->input(0);
output = interp->output(0);
}
// Returns the class index and sets confidence; -1 if uncertain.
int classify(camera_fb_t *fb, float &confidence) {
// Resize/crop fb to IMG×IMG and quantise into input (details omitted).
// ... (image scaling into input->data.int8) ...
if (interp->Invoke() != kTfLiteOk) return -1;
float os = output->params.scale;
int oz = output->params.zero_point;
int best = 0; float bestP = -1;
for (int i = 0; i < N_CLASSES; i++) {
float p = os * (output->data.int8[i] - oz);
if (p > bestP) { bestP = p; best = i; }
}
confidence = bestP;
if (bestP < CONF_THRESHOLD) return -1; // uncertain → human
return best;
}
void setup() {
Serial.begin(115200);
// cameraBegin() as in the doorbell project (PSRAM enabled) ...
inferenceBegin();
Serial.println("Disease camera — flags for confirmation, does not diagnose");
}
void loop() {
camera_fb_t *fb = esp_camera_fb_get();
if (!fb) { delay(1000); return; }
float conf;
int cls = classify(fb, conf);
esp_camera_fb_return(fb);
if (cls < 0) {
Serial.printf("UNCERTAIN (%.2f) — flag for human review\n", conf);
// uplink an "uncertain" flag with the image
} else if (IS_DISEASE[cls]) {
Serial.printf("FLAG: %s (%.2f) — confirm with a person\n", CLASSES[cls], conf);
// uplink a disease flag with the image
} else {
Serial.printf("healthy: %s (%.2f)\n", CLASSES[cls], conf);
}
delay(60000); // inspect once a minute (or on trigger)
}
Configuration & Calibration
Dataset, Model & Training
Dataset
The PlantVillage dataset is the standard starting point: about 54,000 labelled leaf images across 14 crops and 26 diseases, freely available. Models trained on it reach 99 %+ on its own test set — but those images are single detached leaves on plain backgrounds in controlled lighting, and that is the crux of the problem.
A model trained only on PlantVillage typically drops from 99 % to below 50 % on real field images, because the field image (a leaf among clutter, in variable light) looks nothing like the training data. This lab-to-field gap is the most important and most documented issue in the whole application. Do not report the lab accuracy as if it were the field accuracy.
The path to real-world performance is: train on PlantVillage for the base, apply strong augmentation, then fine-tune on your own field images of your crops and diseases. Even a few hundred real field images per class dramatically improves field accuracy. Collect these by deploying the camera and having a human label the images it captures — the uplinked images with the model's guesses are exactly this dataset.
| Dataset | Size | Licence | Use here |
|---|---|---|---|
| PlantVillage | 54,305 images, 38 classes | CC0 / open | Base training — but lab conditions, needs field fine-tuning. |
| PlantDoc | 2,598 field images, 27 classes | CC BY-SA 4.0 | Real field images — much closer to deployment conditions. |
| Your own field images | You collect | Yours | The decisive dataset — your crops, your field, your camera. |
Data preprocessing
- Resize to the model input size (224×224 for MobileNet), preserving aspect ratio where possible, and normalise per the base model's expected input.
- Apply strong augmentation during training: flips, rotation, zoom, brightness, contrast, translation — this is the single most effective lab-to-field mitigation.
- Balance the classes — PlantVillage has uneven class counts, and an imbalanced model over-predicts common classes.
- Hold out a genuinely field-condition test set (PlantDoc or your own field images) and report accuracy on THAT, not on the lab test set.
Model architecture
MobileNetV2 (or V3, or EfficientNet-Lite) via transfer learning is the standard choice for edge deployment: efficient depthwise-separable convolutions, ImageNet-pretrained features, and quantisation-friendly. The base is frozen initially and fine-tuned later.
For the ESP32-S3, the model is quantised to int8 (~1–2 MB) and runs on the vector unit. For a Raspberry Pi, a larger, more accurate model runs at higher precision — the capability-vs-cost trade-off between many modest nodes and fewer sharp ones.
Evaluation, Metrics & Deployment
Figures below distinguish lab and field accuracy deliberately, because the gap between them is the whole story of this application. Reporting only the lab number would be the central dishonesty to avoid.
| Metric | Value | What it tells you |
|---|---|---|
| Lab (PlantVillage) accuracy | 99.2 % | On the clean test set — impressive and misleading if quoted alone. |
| Field accuracy (no field training) | ~45 % | The same model on real field images — the lab-to-field gap in full. |
| Field accuracy (+ augmentation) | ~62 % | Strong augmentation alone recovers a substantial part of the gap. |
| Field accuracy (+ field fine-tuning) | ~85 % | Fine-tuning on real field images closes most of it — the deployment target. |
| Edge inference (ESP32-S3) | ~350 ms | Per image, int8 on the vector unit — fine for periodic inspection. |
| Low-confidence rate (field) | ~15 % | Fraction routed to human review — the honest handling of uncertain cases. |
Figures from the reference training run described above — reproduce them before trusting your own changes.
Testing Procedure & Expected Output
Test from the bottom up. Confirm power, then each sensor in isolation, then the integrated loop — the first failing step tells you exactly where to look.
| Test | What you should see |
|---|---|
| Capture a sharp, well-lit leaf image | The image is in focus, evenly lit, with the leaf filling the frame — the prerequisite for the model to work. |
| Classify a clear healthy leaf of a trained crop | Correct "healthy" class with high confidence. |
| Classify a clearly diseased leaf of a trained disease | The correct disease flagged, with the image logged for confirmation. |
| Show a crop or disease outside the scope | Low confidence → "uncertain" → routed to human, NOT a confident wrong class. |
| Compare lab-test vs field-image accuracy | A large gap without field training — confirming why field data and augmentation matter. |
| Show a nutrient deficiency that mimics a disease | Possibly misclassified — demonstrating the limit and the need for human confirmation. |
| Measure edge inference time | A few hundred milliseconds on the ESP32-S3 — fine for periodic inspection. |
| Deploy, collect and label field images | A field dataset accumulates for fine-tuning, improving accuracy over time. |
Bench-test checklist. If a row fails, stop and fix it before moving on.
Expected output
With everything wired and the firmware uploaded, the Serial Monitor at 115200 baud should look similar to the trace below. Values will differ; the shape of the output should not.
Troubleshooting: Common Errors & Fixes
Performance Optimisation
- Quantise to int8 and run on the ESP32-S3 vector unit for efficient edge inference, or use a Pi for a larger model — choose per your accuracy and power needs.
- Inspect on a schedule or trigger, not continuously — disease develops over days, and periodic inspection is ample and saves power.
- Uplink the image with each flag: it enables remote confirmation and builds the field dataset that improves the model over time.
- Replace every
delay()with amillis()comparison — blocking delays are the single most common cause of dropped readings. - Sample sensors on a fixed cadence and publish on a slower one; you almost never need to transmit at the sampling rate.
- Move networking into its own FreeRTOS task so a slow DNS lookup cannot stall the control loop.
- Use
uint8_t/uint16_twhere the range allows; on an 8-bit AVR a 32-bit add costs four times as much. - For battery builds use deep sleep between samples: an ESP32 drops from ~160 mA awake to about 10 µA asleep, which is the difference between days and months of runtime.
- Profile before optimising — print
micros()deltas around each stage and fix the slowest one first.
Safety Precautions
- This is a screening tool that flags plants for human confirmation. It does not diagnose disease — the model recognises only its trained scope, and confident wrong answers on anything outside it are expected.
- Do not make significant crop-management decisions (spraying, destroying plants) on the model's word alone; confirm flags with a person who can distinguish disease from deficiency and pest damage.
- Report field accuracy honestly; a system marketed on its lab accuracy will disappoint and could lead to wrong decisions.
- Follow safe practice around any agrochemicals applied in response to detections.
- Wear eye protection when soldering or cutting, and solder in a ventilated space — rosin flux fumes are a respiratory irritant.
- Power the circuit through a bench supply with a current limit while you are testing. A 300 mA limit turns a wiring mistake into a beep instead of a dead board.
- Disconnect power before changing any wiring. Hot-plugging a sensor onto a live bus is the fastest way to lose a controller.
Maintenance
- Re-check every screw terminal and header after the first week — thermal cycling loosens connections that felt tight on day one.
- Recalibrate at the interval given in the calibration section, and keep the constants in a text file next to the firmware — not only in flash.
- Keep a short logbook of firmware versions and what changed. Six months later you will not remember why that constant is 1.083.
Future Improvements & Upgrades
A working v1 is a platform, not a finish line. These are the upgrades that add the most capability for the least rework.
- Add field fine-tuning as standard — deploy, collect labelled field images, retrain, and watch field accuracy climb.
- Add an explicit "unknown" class and better out-of-distribution detection so off-scope inputs are handled gracefully.
- Add object detection (locate and classify multiple leaves/lesions in one image) rather than whole-image classification.
- Add severity estimation — not just presence but how much of the leaf is affected, for treatment decisions.
- Add a robotic or rail-mounted platform so one good camera inspects a whole crop rather than needing many fixed nodes.
- Design a proper PCB. Once the breadboard version has run for a month, moving to a two-layer board removes the intermittent-contact failures that dominate prototype faults.
- Add connectivity — an ESP32 and an MQTT publish turn a local gadget into something you can graph, alert on and analyse over months.
- Add persistent local storage (microSD or the on-chip flash) so a network outage does not create a hole in your data.
- Move configuration out of the source: a captive-portal setup page or a JSON config file makes the build reusable without a recompile.
- Add a battery and solar option so the unit survives a power cut and can be sited away from a socket.
- Write a small test harness that feeds synthetic sensor values through the decision logic, so you can validate thresholds without physically triggering the event.
Frequently Asked Questions
References & Learning Resources
These are the primary sources worth reading in full. Manufacturer datasheets always outrank forum posts when the two disagree.
- Hughes & Salathé, "An open access repository of images on plant health (PlantVillage)"arXiv:1511.08060
- Mohanty et al., "Using Deep Learning for Image-Based Plant Disease Detection"Frontiers in Plant Science, 2016
- Singh et al., "PlantDoc: A Dataset for Visual Plant Disease Detection"ACM CoDS-COMAD, 2020
- Barbedo, "Impact of dataset size and variety on the effectiveness of deep learning plant disease detection"Computers and Electronics in Agriculture, 2018
- Sandler et al., "MobileNetV2: Inverted Residuals and Linear Bottlenecks"arXiv:1801.04381
- TensorFlow Lite for Microcontrollers — deployment guideGoogle AI Edge