What is RAG (retrieval-augmented generation)?

RAG, short for retrieval-augmented generation, is a technique that gives a language model relevant information at the moment it answers, rather than relying only on what it learned during training. The system retrieves documents that relate to the question, places them into the model's context, and asks the model to answer using that material. The result is an answer grounded in your data, with a path back to the source.

The motivation is simple. A model trained months ago does not know your current contracts, your latest tickets, or your internal procedures, and asking it to answer from memory invites confident but wrong responses. The original RAG paper introduced the idea of combining a retriever with a generator so the model could pull in non-parametric knowledge on demand. That pattern has since become one of the most common ways to put a model to work on real, proprietary information.

Why retrieval beats memory for operational work

A model's pretrained knowledge is broad, frozen, and unattributable. It cannot cite where a fact came from, it does not update when your data changes, and it has no view of anything private. For operational tasks, those are disqualifying limits. A support agent needs the customer's current entitlements. A claims agent needs the policy as it reads today. A research assistant needs the specific filing in front of it.

Fine-tuning can teach a model style and format, but it is a poor way to load it with facts that change. Retrieval is the better tool for changing facts: update the underlying documents and the next answer reflects them, with no retraining. This is why most teams reach for RAG before they reach for fine-tuning when the problem is "the model needs to know our stuff".

How a RAG pipeline works

A typical RAG system has two phases. Indexing happens ahead of time. Retrieval and generation happen at query time.

The steps behind that diagram:

1. Chunking. Source documents are split into passages small enough to be precise but large enough to stay meaningful. Chunk size is a real design decision, not a default.
2. Embedding. Each chunk is converted into a vector that captures its meaning, then stored in a vector database alongside the original text and metadata.
3. Retrieval. At query time, the question is embedded and compared against the stored vectors to find the most relevant chunks. Many systems add keyword search alongside vector search, an approach called hybrid retrieval.
4. Generation. The retrieved chunks are placed into the model's context with the question, and the model answers using that grounded material, ideally citing which chunk supported each claim.

A reranking step often sits between retrieval and generation, using a more precise model to reorder candidates so the best passages land at the top of the context.

RAG, tools, and MCP

RAG is closely related to the broader idea of giving a model access to context. Retrieval is, in effect, a specialized tool: a "search the knowledge base" capability the system invokes before answering. Seen that way, RAG fits naturally into the agent picture described in what are agent tools.

The Model Context Protocol makes this connection explicit. An MCP server can expose documents as resources or expose a retrieval tool that an agent calls when it needs grounding. In an agent system, retrieval is rarely the whole job; it is one step the agent chooses among others. The model might retrieve, reason, call another tool, and retrieve again. That blend of retrieval and action is the heart of an agentic workflow.

Where RAG goes wrong

RAG removes some failure modes and introduces others. The honest version of the pitch names both.

• Retrieval misses. If the right passage is never retrieved, the model cannot use it. Poor chunking, weak embeddings, or a query that does not match the document's wording all cause this. Hybrid search and reranking help.
• Stale or duplicated sources. RAG is only as current as the index. An out-of-date or conflicting document will be retrieved and trusted just like a good one. Index freshness is an ongoing operational task, not a one-time load.
• Grounded-sounding hallucination. A model can still invent detail even with good context, or blend a retrieved fact with a fabricated one. Requiring citations and checking that claims trace to a real chunk is the defense.
• Prompt injection through retrieved content. If retrieved documents contain instructions, a naive system may follow them. Retrieved content should be treated as data to reason over, never as commands to obey. This matters most when documents come from outside your control.

These are manageable, but only if the team measures them. The same discipline used to evaluate agents applies here: build a set of real questions with known good answers and sources, then track retrieval quality and answer accuracy, often with an LLM-as-a-judge scoring whether each answer is faithful to its sources, as the corpus and prompts change.

When to use RAG, and when not to

RAG is the right tool when answers must be grounded in a body of text that changes over time and when attribution matters: support knowledge bases, policy and contract lookup, internal documentation, research over filings, and customer-specific records. It shines wherever "show me where that came from" is a requirement.

It is the wrong tool when the task does not depend on retrievable facts. Pure reasoning, formatting, classification, or transformation jobs do not need a retriever, and bolting one on adds cost and latency for nothing. For knowledge that almost never changes and is small, putting it directly in the prompt can be simpler than a full pipeline. And for teaching a model a consistent style or output format, fine-tuning remains the better fit. RAG is a precise tool for a specific problem, not a default for every AI feature.

How XY Space approaches it

In XY Space deployments, retrieval is usually one capability inside a larger system rather than the product itself. The agent that handles a workflow may retrieve a policy, extract structured fields, check a rule, and route an edge case to a person, with retrieval grounding only the steps that need it. Sensitive sources stay behind authentication and are filtered at the boundary, retrieved content is treated as untrusted data, and every answer carries its sources so a reviewer can verify it.

Grounding a model in your data is one of the most valuable things you can do with AI, but it rewards careful engineering over a quick demo. If you want help designing a retrieval system that holds up on real documents, talk to us.

Sources

• Lewis et al.: Retrieval-Augmented Generation for Knowledge-Intensive NLP
• Model Context Protocol introduction