Blog · 12 November 2026
Retrieval-augmented generation, or RAG, is a way of answering questions from your own documents. Before the AI writes anything it searches those documents, picks the passages that match the question, and writes the answer from them with the source shown. Because the model reads real text instead of relying on memory, it invents far less.
Why a chatbot makes things up
A language model predicts the next words. It does not look anything up. Ask it about your refund policy and it will produce something that reads like a refund policy, which may be three years old, may be a competitor's, or may be invented. That is not a defect in one model. It is how the model works.
The usual fix is to write longer instructions: be accurate, do not guess, use only the facts below. It helps a little and fails in the same place, because the model still has nowhere to look. The answer has to come from somewhere.
What retrieval adds
- Your documents are split into passages and stored with a searchable fingerprint of their meaning.
- A question arrives and is turned into the same kind of fingerprint.
- The system finds the passages closest to the question, both by meaning and by exact words.
- Only those passages go into the prompt, so the model writes from evidence rather than memory.
- The answer comes back with each claim linked to the passage it came from, so a person can check it in seconds.
That is the whole idea. Retrieval is not a smarter model, it is a smaller question: instead of asking a model what it knows about everything, you ask it what this page says, and it is much better at that.
What good RAG looks like
Most of the work is not the model. It is the pipeline around it.
- A citation on every claim, not a list of links at the bottom of the page.
- An honest refusal. If the documents do not contain the answer, it says so instead of guessing.
- Permissions. A person only gets answers from documents they are allowed to read.
- Freshness. New and changed documents are picked up without rebuilding anything.
- Hybrid search. Meaning finds the paraphrase, keywords find the product code. You need both.
- A test set. Thirty or forty real questions with known answers, run before every change.
When RAG is the wrong tool
RAG answers questions from text. It is the wrong choice in three common cases.
- The answer is a number that changes today, like stock levels, balances or open tickets. That is a database query, not a document search.
- The job is an action, like issuing a refund, booking a slot or opening a ticket. That is a workflow with permissions, not a chat answer.
- You need arithmetic over a table. Retrieval gives you the rows, but something still has to add them up.
A useful assistant usually does all three. It searches documents, queries the database and calls tools. Keel is built that way, so the agent decides which one a question needs.
How to test a RAG system in 20 minutes
- Write down ten questions you already know the answer to, and the document each answer lives in.
- Ask them. Count how many answers are right and how many cite the right document.
- Ask three questions your documents cannot answer. Every one should get a refusal, not a guess.
- Change one document and ask again. If the answer does not move, indexing is broken.
You will learn more from those twenty minutes than from a month of demos. Anyone selling a document assistant should be happy to sit through it.
Where Infrenic uses it
Every assistant we build runs on Keel, which does retrieval on the customer's own documents, in the customer's own cloud, with sources shown on every answer.
See the document assistant we build. Policies, manuals, contracts and past tickets, with an answer and a source.
Read how Keel implements retrieval. Chunking, local embeddings, hybrid search and the rules the agent must follow.
FAQ
What does RAG stand for?
Retrieval-augmented generation. Retrieval finds the relevant passages in your documents, and generation is the model writing an answer from them. The retrieval happens before the model writes, which is what keeps the answer tied to real text.
Is RAG better than fine-tuning?
They do different jobs. RAG keeps knowledge in documents you control and updates it by replacing a file. Fine-tuning changes how a model behaves, its style, format or narrow skill, and is a poor way to add facts because the facts are hard to update and impossible to cite.
Do we need a vector database?
You need somewhere to store and search embeddings. Postgres with the pgvector extension is enough for most companies, including archives of hundreds of thousands of pages. A dedicated vector database makes sense at a scale where that search is the bottleneck.
How much does a RAG assistant cost to run?
The model cost per answer is usually a few cents or less. The real costs are cleaning up the documents once, and testing and improving the answers over time. Any estimate that only counts tokens is missing most of the work.
Infrenic builds AI agents on your data and tools, ships them to production and runs them, all on Keel. Send us a problem and see it working in 48 hours.
The Infrenic AI Dispatch
Practical architecture notes on AI agents, RAG, and production systems. No buzzwords, no spam. Sent when we have something real to share.
Read earlier editions at infrenic.beehiiv.com