What is RAG? Explained with an open-book exam
RAG (retrieval-augmented generation) makes an AI model look up the right pages in your own documents before it answers. Explained with an open-book exam instead of answering from memory.
- Author
Umair Mateen- Published
- Reading time
- 5 min
- Category
- AI
In one sentence: retrieval-augmented generation (RAG) is a way of making an AI model look up the relevant pages in your own documents first, and then write its answer from those pages instead of from memory alone.
The everyday picture
Think of two kinds of exam. In a closed-book exam, the student answers from memory. They may remember a lot, but some details are fuzzy, and anything new since they studied is missing. When they are unsure, they may still write something that sounds right.
In an open-book exam, the student is allowed to bring the textbook. They read the question, turn to the right pages, and write the answer from what is printed there. They can even note which page they used.
A plain AI model sits a closed-book exam every time you ask it something. RAG turns it into an open-book exam, and you choose the book: your price list, policies, manuals or past tickets.
Instead of answering from memory, the system first searches your documents (retrieval), lays the best passages next to the question, and only then lets the AI model write the answer.
How it works, step by step
- Preparing the book (adding tabs and an index). Before anyone asks anything, your documents are cut into short passages called chunks. Anthropic describes chunks as usually a few hundred tokens long. Each one is turned into a list of numbers, called an embedding, that captures its meaning, and stored in a searchable index.
- The question (reading the exam paper). A person asks something, such as "Can I return a sale item?"
- Retrieval (turning to the right pages). A search step finds the chunks closest in meaning to the question. Google Cloud notes that many systems mix meaning-based search with plain keyword search, then rank the results so the most useful pages come first.
- Augmentation (laying the open pages on the desk). Those chunks are pasted into the request next to the question, with an instruction such as "answer using only these passages".
- Generation (writing the answer). The large language model writes a reply based on the pages in front of it. Many setups also show which document each part came from, so a person can check.
Where the picture stops
- The student does not truly understand the book. The model still predicts likely words. Having the right pages makes a correct answer much more likely, but it can still misread or mix up what it was given.
- It only sees the pages it turned to. A real student can flip around until they find the answer. If the search step picks the wrong passages, the model answers from the wrong pages. Google Cloud warns that irrelevant retrieved information can give an answer that is grounded but off-topic or wrong.
- Torn-out pages lose their context. A chunk that says "revenue grew 3%" may not say which company or which quarter. Anthropic points out that this kind of chunking often loses context, which makes retrieval harder.
Why it matters for your business
- Answers from your facts, not the internet’s. The model can quote your current refund rule instead of a general guess.
- Easier to keep up to date. When a policy changes, you update the document. You do not need to retrain the model. The original RAG research paper named updating a model’s knowledge as one of the problems this approach helps with.
- Answers people can check. Showing the source passage lets staff and customers verify an answer, which lowers the risk of a confident mistake going unnoticed.
- Control over who sees what. Because the documents stay in your own store, you can limit which files are searched for which users. This has to be designed in; it does not happen by itself.
An example
Imagine a 30-person furniture shop whose staff answer delivery and warranty questions all day. The rules live in a dozen PDFs that change every season.
The shop sets up an assistant that searches those PDFs before every answer. A staff member asks, "What is the warranty on outdoor chairs?" The assistant finds the current warranty page, drafts a reply from it, and shows a link to the page. The staff member checks it in seconds and sends it. When the warranty changes, the shop swaps in the new PDF, and the answers change with it.
Frequently asked questions
What does RAG stand for?
RAG stands for retrieval-augmented generation. "Retrieval" is the search step, "augmented" means the found passages are added to the request, and "generation" is the AI writing the answer.
Does RAG stop AI hallucinations?
It reduces them but does not remove them. The model can still misread a passage, and if the search finds the wrong pages, the answer can still be wrong. Keep a person checking anything important.
Is RAG the same as training a model on my data?
No. Training or fine-tuning changes the model itself. RAG leaves the model as it is and hands it the relevant pages each time someone asks, so updating the documents updates the answers.
Do I always need RAG?
Not always. If your material is short enough, you can simply include all of it in every request. Anthropic suggests this for knowledge bases under about 200,000 tokens, which is roughly 500 pages. RAG helps when there is too much to send every time.
Related explainers
- What is an LLM? Explained with a new hire who read the whole library
- What is an API? Explained with a waiter in a restaurant
- What is caching? Explained with a restaurant kitchen
The DevDrift builds AI assistants that answer from your own documents, with sources shown and approvals where they matter: see how.