Document Q&A (RAG).

Project Overview

Chat with your own PDFs and documents — retrieval-augmented generation grounds an LLM in your files so answers cite sources instead of hallucinating.

Large language models are fluent and knowledgeable, but they have two problems that make them unreliable for answering questions about your documents: they don't know your private files, and they hallucinate — confidently inventing plausible-sounding answers when they don't actually know. Retrieval-Augmented Generation (RAG) is the technique that fixes both, and it has become the dominant pattern for building useful LLM applications. This project builds a Document Q&A system with RAG: you point it at your own PDFs and documents, ask questions in natural language, and get answers grounded in — and citing — your actual files, not the model's imagination.

The idea is to retrieve first, then generate. Your documents are split into chunks and each is turned into an embedding (a vector capturing its meaning) stored in a vector database. When you ask a question, the system embeds the question, retrieves the most relevant chunks by semantic similarity, and hands them to the LLM as context with an instruction: "answer using only this information, and cite it". The model then generates an answer grounded in the retrieved passages — so it draws on your documents rather than its parametric memory, and can point to which passage each claim came from.

The value is trustworthy question-answering over private knowledge — manuals, contracts, research, wikis — with citations you can verify, and no need to retrain a model when documents change (just update the index). It is honest that RAG is not magic: answer quality is bounded by retrieval quality (if the right chunk isn't retrieved, the answer suffers), chunking and embedding choices matter, the model can still hallucinate or misread even grounded, and it should answer "I don't know" when the documents don't contain the answer. Built well — good chunking, solid retrieval, grounded prompting with citations, and honest "not found" behaviour — it is both the single most useful LLM-application pattern and a complete lesson in composing retrieval and generation.

What this project does

• Answers natural-language questions over your own documents
• Grounds answers in retrieved passages (not model memory)
• Cites which document/passage each answer came from
• Indexes documents as embeddings in a vector database
• Retrieves the most relevant chunks per question
• Updates by re-indexing — no model retraining
• Says "I don't know" when the documents lack the answer

Real-World Applications

Deployment contexts where a build of this kind earns its keep.

Features & Capabilities

• Retrieval-augmented generation (retrieve → generate)
• Document chunking + embedding + vector search
• Grounded, cited answers
• Private-document Q&A
• Index updates without retraining
• Honest "not found" behaviour
• Aware of retrieval-quality and hallucination limits

Difficulty, Time & Required Skills

Skills you should have (or will pick up)

• Document chunking and embedding
• Vector databases and semantic retrieval
• Grounded prompting (context + citations)
• RAG pipeline composition
• Evaluating retrieval and answer grounding

Bill of Materials

Every part below is commonly available from Indian and international hobby-electronics suppliers. Prices are indicative 2026 retail figures in Indian rupees and will drift — treat them as a budgeting guide, not a quotation.

Estimated total: ₹0, excluding tools, shipping and consumables.

Tools and consumables

• Soldering iron (temperature controlled, 350 °C) with 0.8 mm 60/40 or lead-free solder
• Digital multimeter — continuity, DC volts and current ranges
• Wire strippers, flush cutters and a small set of precision screwdrivers
• Heat-shrink tubing and a heat gun (or a lighter, carefully)
• A laptop with a USB port and the toolchain listed above

Hardware Specifications

This is a pure-software system — there is no electronic hardware to specify. The "platform" is a computer plus a language model: a CPU suffices for embedding and vector search, while generation runs on a GPU (local LLM) or a hosted LLM API.

Memory and storage scale with the size of the document corpus and its embedding index; a server deployment adds the chat UI and the vector database service. Everything else is the software stack, models and libraries below.

Software Requirements & Development Environment

Reference toolchain: Python 3.11 + LLM / vector DB. Anything newer normally works; anything older may lack the board definitions used here.

• Install the Arduino IDE 2.3.x (or PlatformIO if you prefer a real editor and dependency locking).
• Add https://espressif.github.io/arduino-esp32/package_esp32_index.json under File → Preferences → Additional Board Manager URLs, then install esp32 from the Boards Manager.
• Set the correct port under Tools → Port. On Linux add yourself to the dialout group: sudo usermod -aG dialout $USER and log out and back in.
• Open the Serial Monitor at 115200 baud — every sketch here logs its state there.
• Keep File → Preferences → Show verbose output during: compilation switched on while you are debugging build errors.

Required libraries

Block Diagram

The block diagram shows the functional decomposition of the system — what senses, what decides, what acts, and where the data ends up.

Circuit Diagram & Wiring

The "wiring" is the RAG data flow — documents are chunked and embedded into a vector store; a question is embedded, relevant chunks retrieved, and passed to the LLM to generate a grounded, cited answer.

Wire one row at a time and tick it off — most "it does not work" reports trace back to a single swapped pair.

Wiring explanation

• Split documents into chunks and embed each into a vector database (indexing, done once/updated).
• Embed the user question and retrieve the most relevant chunks.
• Pass the retrieved chunks to the LLM as context with a grounded, cite-your-sources instruction.
• Return the answer with citations; answer "I don't know" if unsupported.
• Answer quality depends on retrieval — invest in chunking/embedding/search.

System Architecture

Read the stack from the bottom up: physical hardware, the firmware that drives it, the transport that moves data off the device, and the software a human actually looks at.

Working Principle

A raw LLM answering questions about your documents fails in two specific ways, and RAG is engineered precisely against them. First, the model does not contain your private documents — its knowledge is whatever it was trained on, which is not your contracts or manuals. Second, and worse, when asked something it does not know, an LLM tends to hallucinate: it produces a fluent, confident, wrong answer, because it is optimised to generate plausible text, not to know its own limits. Retrieval-Augmented Generation addresses both by changing where the model gets its facts: instead of relying on its parametric memory, it is fed the relevant passages from your documents and instructed to answer from them. The model's fluency is kept; its unreliability as a knowledge source is bypassed.

The architecture is retrieve, then generate, and the indexing half comes first. Documents are split into chunks (passages small enough to be specific but large enough to be meaningful), and each chunk is converted into an embedding — a vector capturing its meaning — and stored in a vector database. This index is built once and updated when documents change; crucially, adding or changing documents needs no model retraining, just re-indexing, which is a major practical advantage over trying to bake knowledge into a model.

At query time, the retrieval step is the heart of the system. The question is embedded into the same vector space, and the database returns the most semantically similar chunks — the passages most likely to contain the answer. These retrieved chunks are then inserted into the LLM's prompt as context, with an instruction to answer using only the provided information and to cite which passage supports each claim. The model now generates grounded in your documents: its answer is anchored to real retrieved text, and because it cites sources, a user can verify each claim against the original — the antidote to unverifiable hallucination.

The honesty RAG demands is that it is not magic, and its weakest link is retrieval. The generator can only be as good as what it is given: if the relevant chunk is not retrieved (bad chunking, weak embeddings, an ambiguous question), the answer will be incomplete or wrong no matter how capable the LLM — "retrieval quality bounds answer quality" is the governing principle, which is why chunk size, embedding choice and search matter so much. Even with good retrieval, the model can still misread the context or hallucinate beyond it, so grounding is a strong mitigation, not a guarantee — which makes citations (so claims are checkable) and honest "I don't know" behaviour (when the documents genuinely lack the answer, the system must say so rather than invent) essential parts of the design, not niceties. Built with those principles — thoughtful chunking, solid semantic retrieval, grounded-and-cited prompting, and a willingness to admit ignorance — RAG turns an unreliable know-it-all into a trustworthy assistant over your own knowledge, which is exactly why it has become the default pattern for real LLM applications.

The maths behind it

Indexing

for each document:
  chunks = split(document)         # meaningful passages
  for c in chunks: store(embed(c), c)  # vector DB

Built once; update by re-indexing. NO model retraining.

Retrieval (the crux)

q = embed(question)
top_k = vector_db.search(q, k)      # most similar chunks

Answer quality ≤ retrieval quality: if the right chunk
isn't retrieved, the answer suffers.

Grounded generation

answer = LLM(prompt = "Answer using ONLY:\n" + top_k +
              "\nQuestion: " + question +
              "\nCite sources. If not supported, say I don't know.")

Grounded in your docs + cited → verifiable, not hallucinated.

Program Flowchart

The firmware is a single cooperative loop. Nothing blocks for long, so networking, sensing and the user interface all stay responsive.

Assembly Instructions

Build on a breadboard first and only commit to solder once the whole system has run for an hour without a fault.

1. Index your documents

   Chunk documents, embed each chunk with source metadata, and store them in a vector database.

2. Retrieve and ground

   Embed the question, retrieve the most relevant chunks, and prompt the LLM to answer using only them and cite sources.

3. Cite and handle "not found"

   Return citations for verification, and answer "I don't know" when retrieval finds nothing relevant.

Step-by-Step Implementation Guide

Work through these in order. Each step ends in something you can observe, so a failure is always localised to the step you just finished.

1. Retrieve relevant chunks, then generate grounded

   Embed the question, retrieve top chunks, and have the LLM answer using only them with citations — or admit it does not know.

   pythonrag.py Copy Download

   MIN_SIM = 0.25
   
   def answer(question, embed, db, llm, k=5):
       chunks = db.search(embed(question), k=k)      # RETRIEVE first
       if not chunks or chunks[0].score < MIN_SIM:
           return {"answer": "I don't know based on these documents.", "sources": []}
       context = "\n\n".join(f"[{c.source} p{c.page}] {c.text}" for c in chunks)
       prompt = ("Answer using ONLY the context and cite sources [file pN]; "
                 "if unsupported, say you don't know.\n\n"
                 f"{context}\n\nQ: {question}")
       return {"answer": llm.generate(prompt),        # GENERATE grounded + cited
               "sources": [(c.source, c.page) for c in chunks]}

   chunks = db.search(embed(question), k=k) # RETRIEVE firstRetrieval comes first — the question is matched against the indexed chunks to find the passages likely to hold the answer.

   if not chunks or chunks[0].score < MIN_SIM:If nothing relevant is retrieved, the system honestly says it does not know rather than inviting the model to invent an answer.

   prompt = ("Answer using ONLY the context and cite sources [file pN]; "The prompt grounds the model in the retrieved passages and demands citations — the core of trustworthy RAG.

   "sources": [(c.source, c.page) for c in chunks]}Returning the sources lets a user verify every claim against the original document — the antidote to unverifiable hallucination.

2. Cite, verify and iterate on retrieval

   Show citations so answers are checkable, and improve chunking/embeddings/top-k (and add reranking) where retrieval misses.

Complete Source Code

The listing below is complete and compiles as written — there are no elided sections. Read the annotations under each block before you upload it.

#!/usr/bin/env python3
"""
Document Q&A with Retrieval-Augmented Generation (RAG)

Chat over YOUR documents: chunk + embed them into a vector store;
for each question RETRIEVE the most relevant chunks and GENERATE an
answer grounded in them WITH CITATIONS. Answers "I don't know" when
unsupported. Answer quality is bounded by retrieval quality.
"""
MIN_SIM = 0.25

class DocumentQA:
    def __init__(self, embed, vector_db, llm):
        self.embed = embed; self.db = vector_db; self.llm = llm

    def index(self, documents):
        for doc in documents:
            for chunk in split(doc):                  # meaningful passages
                self.db.add(self.embed(chunk.text),   # embedding
                            meta={"source": doc.name, "page": chunk.page,
                                  "text": chunk.text})
        # No model retraining — updates are just re-indexing.

    def ask(self, question, k=5):
        hits = self.db.search(self.embed(question), k=k)   # RETRIEVE
        if not hits or hits[0].score < MIN_SIM:
            return {"answer": "I don't know based on the documents.",
                    "sources": []}                          # honest not-found
        context = "\n\n".join(
            f"[{h.meta['source']} p{h.meta['page']}] {h.meta['text']}" for h in hits)
        prompt = ("Answer the question using ONLY the context below. "
                  "Cite sources as [file pN]. If the answer is not in the "
                  "context, say you don't know.\n\n"
                  f"Context:\n{context}\n\nQuestion: {question}")
        return {"answer": self.llm.generate(prompt),        # GENERATE grounded
                "sources": [(h.meta['source'], h.meta['page']) for h in hits]}

if __name__ == "__main__":
    qa = DocumentQA(embed_model, VectorDB(), LLM())
    qa.index(load_documents("./docs"))
    print(qa.ask("What is the warranty period?"))
    # Debug wrong answers by checking retrieval FIRST — most failures are there.

Configuration & Calibration

Configuration steps

• Configure chunking (size/overlap), the embedding model and vector database.
• Configure retrieval top-k, similarity threshold and optional reranking.
• Configure the LLM and the grounded, cite-your-sources prompt.
• Configure "I don't know" behaviour and citation display.

Calibration procedure

An uncalibrated sensor produces confident, precise, wrong numbers. Do this once per physical unit and record the constants.

1. Retrieval

   Verify the right chunks are retrieved for representative questions; tune chunking/embeddings/top-k.

2. Grounding

   Check answers are supported by and cite the retrieved passages; tighten the prompt.

3. Not-found

   Confirm it says "I don't know" when documents lack the answer.

Dataset, Model & Training

Dataset

Your own documents (PDFs, docs, wikis) are the knowledge source — chunked and embedded, not used to train the model.

An embedding model and an LLM are pretrained; the "data" is your indexed corpus.

Data preprocessing

• Parse documents; split into chunks (size/overlap tuned for specificity vs context).
• Embed chunks; store with metadata (source, page) for citations.
• Clean/normalise text; handle tables/headers where possible.

Hyperparameters

Training process

• No training of the LLM/embedder needed — RAG is composition, not fine-tuning.
• Tune chunking, embeddings, top-k and prompts; optionally add a reranker.
• Evaluate retrieval hit-rate and answer grounding on a Q&A set.

Evaluation, Metrics & Deployment

The key metrics are retrieval quality (did the right chunk come back?) and answer faithfulness (is the answer grounded and cited, with honest "I don't know"?).

Figures from the reference training run described above — reproduce them before trusting your own changes.

Inference example

def answer(question, embed, vector_db, llm, k=5):
    q = embed(question)
    chunks = vector_db.search(q, k=k)          # RETRIEVE relevant passages
    if not chunks or chunks[0].score < MIN_SIM:
        return {"answer": "I don't know based on the documents.",
                "sources": []}                 # honest: docs lack the answer
    context = "\n\n".join(f"[{c.source} p{c.page}] {c.text}" for c in chunks)
    prompt = ("Answer the question using ONLY the context. "
              "Cite sources like [file pN]. If unsupported, say you don't know.\n\n"
              f"Context:\n{context}\n\nQuestion: {question}")
    text = llm.generate(prompt)                 # GENERATE grounded + cited
    return {"answer": text, "sources": [(c.source, c.page) for c in chunks]}
    # Answer quality is bounded by retrieval; grounding reduces, not removes, error.

Testing Procedure & Expected Output

Test from the bottom up. Confirm power, then each sensor in isolation, then the integrated loop — the first failing step tells you exactly where to look.

Bench-test checklist. If a row fails, stop and fix it before moving on.

Expected output

Grounded, cited answers over your documents, with honest "I don't know" when unsupported.

{
  "question": "What is the warranty period?",
  "answer": "The warranty period is 24 months from purchase [manual.pdf p12].",
  "sources": [["manual.pdf", 12]],
  "grounded": true
}

A grounded answer citing the exact page it came from — verifiable against the source document, not an unsupported guess; an out-of-scope question would return "I don't know".

Troubleshooting: Common Errors & Fixes

Performance Optimisation

• Invest in retrieval — it bounds answer quality.
• Tune chunking, embeddings, top-k; add reranking.
• Ground strictly and require citations to reduce hallucination.
• Update by re-indexing; no model retraining needed.
• Replace every delay() with a millis() comparison — blocking delays are the single most common cause of dropped readings.
• Sample sensors on a fixed cadence and publish on a slower one; you almost never need to transmit at the sampling rate.
• Move networking into its own FreeRTOS task so a slow DNS lookup cannot stall the control loop.
• Use uint8_t / uint16_t where the range allows; on an 8-bit AVR a 32-bit add costs four times as much.
• Profile before optimising — print micros() deltas around each stage and fix the slowest one first.

Safety Precautions

• Grounding reduces but does not eliminate hallucination — keep citations so claims are verifiable.
• Answer "I don't know" when documents lack the answer, rather than inventing.
• Private documents are sensitive — secure the index, access and any LLM API use.
• Do not rely on answers for high-stakes decisions without human verification of sources.
• Wear eye protection when soldering or cutting, and solder in a ventilated space — rosin flux fumes are a respiratory irritant.
• Power the circuit through a bench supply with a current limit while you are testing. A 300 mA limit turns a wiring mistake into a beep instead of a dead board.
• Disconnect power before changing any wiring. Hot-plugging a sensor onto a live bus is the fastest way to lose a controller.

Maintenance

• Re-index as documents change; keep the corpus current.
• Monitor retrieval and grounding quality; iterate.
• Update embedding/LLM models as better ones appear.
• Audit access to sensitive documents and answers.
• Re-check every screw terminal and header after the first week — thermal cycling loosens connections that felt tight on day one.
• Recalibrate at the interval given in the calibration section, and keep the constants in a text file next to the firmware — not only in flash.
• Keep a short logbook of firmware versions and what changed. Six months later you will not remember why that constant is 1.083.

Future Improvements & Upgrades

A working v1 is a platform, not a finish line. These are the upgrades that add the most capability for the least rework.

• Add reranking and hybrid (keyword+semantic) retrieval.
• Add multi-hop/agentic retrieval for complex questions.
• Add answer-faithfulness checks/guardrails.
• Add conversational memory over documents.
• Design a proper PCB. Once the breadboard version has run for a month, moving to a two-layer board removes the intermittent-contact failures that dominate prototype faults.
• Add connectivity — an ESP32 and an MQTT publish turn a local gadget into something you can graph, alert on and analyse over months.
• Add persistent local storage (microSD or the on-chip flash) so a network outage does not create a hole in your data.
• Move configuration out of the source: a captive-portal setup page or a JSON config file makes the build reusable without a recompile.
• Add a battery and solar option so the unit survives a power cut and can be sited away from a socket.
• Write a small test harness that feeds synthetic sensor values through the decision logic, so you can validate thresholds without physically triggering the event.

Frequently Asked Questions

References & Learning Resources

These are the primary sources worth reading in full. Manufacturer datasheets always outrank forum posts when the two disagree.

1. Retrieval-augmented generationReference
2. Vector databaseReference
3. Sentence embeddingsReference
4. LLM hallucinationReference
5. FAISS similarity searchLibrary