Abstractive Text Summarizer.

Project Overview

Condenses long documents into crisp, readable summaries — writing new sentences that capture the essence, not just pasting extracts.

There is far more to read than anyone has time for — reports, articles, threads, transcripts — and the ability to condense a long document into a short, faithful summary is one of the most broadly useful things NLP can do. This project builds an abstractive summarizer: given a long document, it produces a crisp, readable summary that captures the essence. Crucially, it is abstractive — it writes new sentences in its own words, the way a person would, rather than merely stitching together sentences copied from the source.

That abstractive-vs-extractive distinction is the heart of the project. Extractive summarization selects and concatenates the most important existing sentences — safe (every sentence is verbatim from the source) but often choppy and limited. Abstractive summarization, powered by sequence-to-sequence transformer models (or LLMs), generates a summary, paraphrasing, compressing and rephrasing to produce something fluent and genuinely concise — much closer to how a human summarizes. This is more powerful and more natural, but it introduces a risk extractive methods don't have.

The value is turning long content into something quickly digestible while preserving meaning. And the honesty is essential and specific: because an abstractive model generates text, it can hallucinate — introduce facts, names or claims that are not in the source — which is uniquely dangerous in a summary, whose entire job is to faithfully represent the original. So faithfulness matters as much as fluency: a summary that reads beautifully but misstates the source is worse than useless. A good summarizer is therefore evaluated not just on readability but on factual consistency with the source, handles very long inputs (chunking), and is used with awareness that the output must be checkable against the original. Built with that faithfulness-first mindset, it is both a genuinely useful tool and a clear lesson in generative NLP and its central risk.

What this project does

• Condenses long documents into short summaries
• Writes new sentences (abstractive, not extractive)
• Paraphrases and compresses like a human summarizer
• Handles long inputs via chunking
• Aims for faithfulness to the source, not just fluency
• Makes long content quickly digestible
• Flags/guards against hallucinated content

Real-World Applications

Deployment contexts where a build of this kind earns its keep.

Features & Capabilities

• Abstractive (generative) summarization
• Sequence-to-sequence / LLM models
• Long-input handling (chunk + combine)
• Faithfulness/factual-consistency focus
• Length/style control
• Readability + conciseness
• Honest about hallucination risk in summaries

Difficulty, Time & Required Skills

Skills you should have (or will pick up)

• Abstractive vs extractive summarization
• Sequence-to-sequence / LLM summarization
• Long-input chunking and combination
• Faithfulness / factual-consistency evaluation
• Length/style control and prompting

Bill of Materials

Every part below is commonly available from Indian and international hobby-electronics suppliers. Prices are indicative 2026 retail figures in Indian rupees and will drift — treat them as a budgeting guide, not a quotation.

Estimated total: ₹0, excluding tools, shipping and consumables.

Tools and consumables

• Soldering iron (temperature controlled, 350 °C) with 0.8 mm 60/40 or lead-free solder
• Digital multimeter — continuity, DC volts and current ranges
• Wire strippers, flush cutters and a small set of precision screwdrivers
• Heat-shrink tubing and a heat gun (or a lighter, carefully)
• A laptop with a USB port and the toolchain listed above

Hardware Specifications

This is a pure-software NLP system — no electronic hardware to specify. The "platform" is a computer plus a summarization model: a GPU (local seq2seq/LLM) or a hosted API handles generation, and a CPU handles chunking and faithfulness checks.

Memory scales with input length and model size; very long documents are chunked to fit the model context. A deployment adds the input/output UI and optional faithfulness-logging. Everything else is the software stack, models and libraries below.

Software Requirements & Development Environment

Reference toolchain: Python 3.11 + transformers / LLM. Anything newer normally works; anything older may lack the board definitions used here.

• Install the Arduino IDE 2.3.x (or PlatformIO if you prefer a real editor and dependency locking).
• Add https://espressif.github.io/arduino-esp32/package_esp32_index.json under File → Preferences → Additional Board Manager URLs, then install esp32 from the Boards Manager.
• Set the correct port under Tools → Port. On Linux add yourself to the dialout group: sudo usermod -aG dialout $USER and log out and back in.
• Open the Serial Monitor at 115200 baud — every sketch here logs its state there.
• Keep File → Preferences → Show verbose output during: compilation switched on while you are debugging build errors.

Required libraries

Block Diagram

The block diagram shows the functional decomposition of the system — what senses, what decides, what acts, and where the data ends up.

Circuit Diagram & Wiring

The "wiring" is the summarization data flow — a long document is (chunked and) passed to an abstractive model that generates a concise summary, which is then checked for faithfulness to the source.

Wire one row at a time and tick it off — most "it does not work" reports trace back to a single swapped pair.

Wiring explanation

• Provide the long document as input.
• Chunk very long inputs to fit the model, then combine partial summaries.
• Generate an abstractive summary (new sentences, not extracts).
• Check the summary for factual consistency with the source.
• Faithfulness matters as much as fluency — a fluent but wrong summary is worse than useless.

System Architecture

Read the stack from the bottom up: physical hardware, the firmware that drives it, the transport that moves data off the device, and the software a human actually looks at.

Working Principle

Summarization comes in two fundamentally different flavours, and understanding the difference is the whole conceptual core of this project. Extractive summarization selects the most important sentences from the source and concatenates them — every word in the summary is verbatim from the original. It is safe (it cannot introduce anything false) but limited: the result is often choppy, redundant, and constrained by whatever sentences happen to exist. Abstractive summarization instead generates a summary in new words — paraphrasing, compressing, merging ideas across sentences — the way a person actually summarizes. It is far more fluent, concise and natural, which is why it is the more powerful and more desirable approach.

Abstractive summarization is made possible by sequence-to-sequence models — transformers that read an input sequence (the document) and generate an output sequence (the summary) — and by LLMs, which are exceptional summarizers. These models have learned, from many document–summary pairs, how to identify what matters and re-express it briefly and coherently. The result reads like human writing rather than a collage of extracts, capturing the essence rather than just the highlights.

But generation brings a risk that extraction, by construction, does not have, and it is the defining hazard of the project: hallucination. Because the model produces new text rather than copying, it can introduce facts, figures, names or claims that are simply not in the source — or subtly distort what the source said. In most generation tasks a small invention is a minor flaw; in a summary it is a fundamental failure, because a summary's entire purpose is to faithfully represent the original. A summary that adds a statistic the document never mentioned, or flips a conclusion, is actively misleading precisely where the reader is trusting it to be accurate.

This is why the guiding principle of a good summarizer is that faithfulness matters at least as much as fluency. A summary that reads beautifully but misstates the source is worse than a clumsy one that is accurate, because it launders error into confident prose. Consequently, a serious summarizer is evaluated on factual consistency with the source — not just readability metrics — and is designed to minimise hallucination (grounding the model in the source, constraining it, and checking the output against the original). Practically, it must also handle very long inputs that exceed the model's context, typically by chunking the document, summarising the parts, and combining them — carefully, since combination can itself introduce errors. And it should offer length/style control and be used with the awareness that its output is a claim about a source that can be checked against it. Built faithfulness-first — abstractive fluency, but disciplined by factual consistency and long-input handling — the summarizer delivers real value (long content made quickly digestible) while teaching the central promise and the central peril of generative NLP.

The maths behind it

Extractive vs abstractive

Extractive:  summary = select+concat(source sentences)
  → verbatim, safe, but choppy/limited

Abstractive: summary = generate(new sentences)
  → fluent, concise, human-like — BUT can hallucinate.

Abstractive generation

summary = seq2seq / LLM (document)

Learned to identify what matters and re-express it briefly.
Reads like human writing, not a collage of extracts.

Faithfulness (the key metric)

faithful if every claim in summary ⊆ information in source

  hallucination = claim in summary NOT supported by source
  goal: fluency AND factual consistency

A fluent-but-wrong summary is worse than a clumsy-but-true one.

Program Flowchart

The firmware is a single cooperative loop. Nothing blocks for long, so networking, sensing and the user interface all stay responsive.

Assembly Instructions

Build on a breadboard first and only commit to solder once the whole system has run for an hour without a fault.

1. Set up abstractive summarization

   Use a seq2seq/LLM model to generate summaries in new words, with length/style control.

2. Handle long inputs

   Chunk very long documents, summarise the parts, and combine carefully.

3. Check faithfulness

   Evaluate the summary's factual consistency with the source, and regenerate/flag if it strays.

Step-by-Step Implementation Guide

Work through these in order. Each step ends in something you can observe, so a failure is always localised to the step you just finished.

1. Summarise abstractively, then check faithfulness

   Generate an abstractive summary (chunking if long) and verify it is factually consistent with the source.

   pythonsummarize.py Copy Download

   FAITHFUL_MIN = 0.8
   
   def summarize(document, model, max_len=150):
       if too_long(document):
           parts = [model.summarize(c, max_len//2) for c in chunk(document)]  # long
           draft = model.summarize(" ".join(parts), max_len)                  # combine
       else:
           draft = model.summarize(document, max_len)     # abstractive: new sentences
   
       score = factual_consistency(draft, document)       # faithfulness, not just fluency
       if score < FAITHFUL_MIN:
           draft = model.summarize(document, max_len, grounded=True)  # tighten to source
       return {"summary": draft, "faithfulness": round(score, 2)}

   parts = [model.summarize(c, max_len//2) for c in chunk(document)] # longVery long inputs are chunked and summarised piecewise, then combined, so documents beyond the model's context can still be handled.

   draft = model.summarize(document, max_len) # abstractive: new sentencesThe summary is generated in new words — abstractive, fluent, human-like — not a collage of extracted sentences.

   score = factual_consistency(draft, document) # faithfulness, not just fluencyThe summary is scored for factual consistency with the source — the metric that actually matters, since a fluent but wrong summary is worse than useless.

   draft = model.summarize(document, max_len, grounded=True) # tighten to sourceIf faithfulness is low the summary is regenerated more tightly grounded in the source, guarding against hallucination.

2. Control length/style and keep it checkable

   Offer length/style control and present the summary as a claim about the source that can be verified against it.

Complete Source Code

The listing below is complete and compiles as written — there are no elided sections. Read the annotations under each block before you upload it.

#!/usr/bin/env python3
"""
Abstractive Text Summarizer

Generates crisp, readable summaries in NEW words (abstractive), not
extracts. Handles long inputs by chunking + combining. Because it
GENERATES text it can hallucinate — so FAITHFULNESS (factual consistency
with the source) matters as much as fluency and is checked, not assumed.
"""
FAITHFUL_MIN = 0.8

class Summarizer:
    def __init__(self, model, checker):
        self.model = model            # seq2seq / LLM (abstractive)
        self.checker = checker        # factual-consistency scorer

    def summarize(self, document, max_len=150, style=None):
        if too_long(document):                         # exceeds model context
            partials = [self.model.summarize(c, max_len // 2)
                        for c in chunk(document, overlap=True)]   # per chunk
            draft = self.model.summarize(" ".join(partials), max_len, style)
        else:
            draft = self.model.summarize(document, max_len, style)  # abstractive

        # FAITHFULNESS FIRST: a fluent but inaccurate summary is worse than none.
        score = self.checker.consistency(draft, document)
        if score < FAITHFUL_MIN:
            draft = self.model.summarize(document, max_len, style,
                                         grounded=True)   # tighten to source
            score = self.checker.consistency(draft, document)
        return {"summary": draft, "faithfulness": round(score, 2)}

if __name__ == "__main__":
    s = Summarizer(AbstractiveModel(), FaithfulnessChecker())
    print(s.summarize(load_text("report.txt"), max_len=120))
    # Evaluate factual consistency, not just readability; verify vs the source.

Configuration & Calibration

Configuration steps

• Configure the abstractive model, target length and style.
• Configure chunking/combination for long inputs.
• Configure the faithfulness/factual-consistency check and threshold.
• Configure grounded regeneration on low faithfulness.

Calibration procedure

An uncalibrated sensor produces confident, precise, wrong numbers. Do this once per physical unit and record the constants.

1. Faithfulness

   Evaluate factual consistency on a held-out set; tune grounding/temperature to reduce hallucination.

2. Length/coverage

   Balance conciseness against covering the key points.

3. Long inputs

   Verify chunking/combination preserves meaning without introducing errors.

Dataset, Model & Training

Dataset

Abstractive models are trained on document–summary pairs; the model is usually pretrained and optionally fine-tuned for domain/length/style.

Faithfulness evaluation uses source–summary consistency checks, not just overlap metrics.

Data preprocessing

• Clean text; chunk long documents to fit the model context (with overlap).
• Optionally set target length/style; segment sections for structured docs.
• Prepare source spans for faithfulness checking.

Hyperparameters

Training process

• Use a strong pretrained abstractive model; fine-tune for domain/length/style if needed.
• Optimise for faithfulness (low-temperature, grounded prompting), not just overlap metrics.
• Evaluate factual consistency on a held-out set, not only ROUGE.

Evaluation, Metrics & Deployment

Readability/overlap (e.g. ROUGE) is necessary but not sufficient — factual consistency with the source is the metric that actually matters for a summary.

Figures from the reference training run described above — reproduce them before trusting your own changes.

Inference example

def summarize(document, model, max_len=150):
    if too_long(document):                     # exceeds model context
        parts = [model.summarize(c, max_len//2)  # summarise each chunk
                 for c in chunk(document)]
        draft = model.summarize(" ".join(parts), max_len)  # combine
    else:
        draft = model.summarize(document, max_len)          # abstractive

    # FAITHFULNESS is the point: a fluent-but-wrong summary is worse than none.
    score = factual_consistency(draft, document)   # claims supported by source?
    if score < FAITHFUL_MIN:
        draft = model.summarize(document, max_len, grounded=True)  # tighten
    return {"summary": draft, "faithfulness": score}
    # Evaluate factual consistency, not just readability.

Testing Procedure & Expected Output

Test from the bottom up. Confirm power, then each sensor in isolation, then the integrated loop — the first failing step tells you exactly where to look.

Bench-test checklist. If a row fails, stop and fix it before moving on.

Expected output

Concise, fluent summaries checked for faithfulness to the source.

{
  "length_words": 118,
  "summary": "The report finds regional demand rose in Q2, driven mainly by ...",
  "faithfulness": 0.91,
  "note": "abstractive; verify claims against the source"
}

A crisp abstractive summary with a high faithfulness score — fluent and genuinely concise, while checked for factual consistency so it represents the source rather than embellishing it.

Troubleshooting: Common Errors & Fixes

Performance Optimisation

• Optimise and evaluate for faithfulness, not just fluency/overlap.
• Ground to the source and use low temperature to cut hallucination.
• Chunk and combine long inputs carefully.
• Offer length/style control; keep output checkable.
• Replace every delay() with a millis() comparison — blocking delays are the single most common cause of dropped readings.
• Sample sensors on a fixed cadence and publish on a slower one; you almost never need to transmit at the sampling rate.
• Move networking into its own FreeRTOS task so a slow DNS lookup cannot stall the control loop.
• Use uint8_t / uint16_t where the range allows; on an 8-bit AVR a 32-bit add costs four times as much.
• Profile before optimising — print micros() deltas around each stage and fix the slowest one first.

Safety Precautions

• Abstractive summaries can hallucinate — never trust a summary of high-stakes content without verifying against the source.
• Faithfulness matters as much as fluency; a fluent but wrong summary is actively misleading.
• Documents may be sensitive — handle securely and lawfully.
• Present summaries as claims about a source that can and should be checked.
• Wear eye protection when soldering or cutting, and solder in a ventilated space — rosin flux fumes are a respiratory irritant.
• Power the circuit through a bench supply with a current limit while you are testing. A 300 mA limit turns a wiring mistake into a beep instead of a dead board.
• Disconnect power before changing any wiring. Hot-plugging a sensor onto a live bus is the fastest way to lose a controller.

Maintenance

• Re-evaluate faithfulness as models/data change.
• Tune chunking/combination for new document types.
• Update the model for better quality/faithfulness.
• Monitor for hallucination in real use.
• Re-check every screw terminal and header after the first week — thermal cycling loosens connections that felt tight on day one.
• Recalibrate at the interval given in the calibration section, and keep the constants in a text file next to the firmware — not only in flash.
• Keep a short logbook of firmware versions and what changed. Six months later you will not remember why that constant is 1.083.

Future Improvements & Upgrades

A working v1 is a platform, not a finish line. These are the upgrades that add the most capability for the least rework.

• Add citation/grounding to source spans in the summary.
• Add query-focused and multi-document summarization.
• Add stronger automatic faithfulness checking.
• Add controllable abstraction level (extractive↔abstractive).
• Design a proper PCB. Once the breadboard version has run for a month, moving to a two-layer board removes the intermittent-contact failures that dominate prototype faults.
• Add connectivity — an ESP32 and an MQTT publish turn a local gadget into something you can graph, alert on and analyse over months.
• Add persistent local storage (microSD or the on-chip flash) so a network outage does not create a hole in your data.
• Move configuration out of the source: a captive-portal setup page or a JSON config file makes the build reusable without a recompile.
• Add a battery and solar option so the unit survives a power cut and can be sited away from a socket.
• Write a small test harness that feeds synthetic sensor values through the decision logic, so you can validate thresholds without physically triggering the event.

Frequently Asked Questions

References & Learning Resources

These are the primary sources worth reading in full. Manufacturer datasheets always outrank forum posts when the two disagree.

1. Automatic summarizationReference
2. Abstractive vs extractiveReference
3. Sequence-to-sequence modelsReference
4. Faithfulness / hallucinationReference
5. ROUGE metricReference