Skip to content

What Is RAG (Retrieval-Augmented Generation)? How It Works and When to Use It

Retrieval-augmented generation (RAG) lets an AI model look up relevant documents before answering, so it can use your own data and cite sources. How RAG works step by step, embeddings and vector databases, RAG vs fine-tuning, and common mistakes.

4 min read

What Is RAG in AI? — cover image from TensorCore.si

Short answer

RAG (retrieval-augmented generation) is a technique that lets a large language model look up relevant information before it answers. When you ask a question, the system searches a collection of documents, such as your company files or a website, finds the most relevant passages and adds them to the prompt. The model then answers using those passages and can cite them. RAG gives AI access to private and up-to-date information, and reduces made-up answers, without retraining the model.

Why RAG is needed

Large language models only know what was in their training data. That causes three problems:

  • No private knowledge: the model has never seen your internal documents, products or policies.
  • Outdated knowledge: training data stops at a cut-off date.
  • Hallucinations: when a model does not know something, it may invent a plausible answer.

RAG solves these by giving the model the right information at the moment it answers. The term comes from a 2020 research paper by Patrick Lewis and colleagues at Facebook AI Research (now Meta AI).

How RAG works, step by step

Preparing the data (done once)

  • Collect documents: PDFs, web pages, help articles, tickets, code.
  • Chunk: split them into passages of a few hundred words.
  • Embed: convert each chunk into an embedding, a vector of numbers that captures its meaning, using an embedding model.
  • Store: save the vectors in a vector database (or a search index) with links back to the original text.

Answering a question (every time)

  • Embed the question with the same embedding model.
  • Retrieve: find the chunks whose vectors are closest in meaning, often combined with keyword search (hybrid search).
  • Rerank (optional): use a second model to pick the best few passages.
  • Augment: insert those passages into the prompt with instructions such as "answer using only these sources and cite them".
  • Generate: the LLM writes the answer.

RAG vs fine-tuning vs long context

ApproachWhat it changesBest forDownsides
RAGWhat the model sees at answer timeFacts that change, private documents, citationsQuality depends on retrieval
Fine-tuningThe model's weightsStyle, format, specialised behaviourCostly to update, can still hallucinate facts
Long contextPuts whole documents in the promptA few long documentsExpensive per question, limited by context size

Many real systems combine them: a fine-tuned or instruction-tuned model, RAG for facts, and a large context window for the retrieved passages.

Common RAG mistakes

  • Bad chunking: chunks that are too big dilute relevance; too small and they lose context.
  • Only using vector search: exact names, codes and numbers are often found better by keyword search, so hybrid search usually works best.
  • No evaluation: test retrieval quality with real questions before blaming the model.
  • Stale index: re-embed documents when they change.
  • No citations: showing sources lets users check answers and builds trust.

RAG and agents

In agentic RAG, an AI agent decides when to search, what to search for and whether the results are good enough, and may run several searches before answering. Web search in chat assistants is a form of RAG over the internet.

What hardware does RAG need?

RAG is lighter than training. You need an embedding model (small, can run on CPU or a modest GPU), a vector database and an LLM for answers. Running everything locally is possible with an open model; see how to run an LLM locally and how much VRAM an LLM needs. Retrieved passages add tokens to every prompt, which increases the KV cache and compute per question.

Frequently asked questions

What does RAG stand for in AI?

Retrieval-augmented generation: retrieving relevant information and adding it to the prompt so the model's generated answer is grounded in it.

Is RAG better than fine-tuning?

For adding knowledge, usually yes, because it is cheaper, easier to update and can cite sources. Fine-tuning is better for changing a model's style or behaviour.

What is a vector database?

A database built to store embeddings and quickly find the vectors most similar to a query. Examples include pgvector (for PostgreSQL), Pinecone, Weaviate, Milvus and Chroma.

Does RAG stop hallucinations?

It reduces them a lot when the right passages are retrieved, but the model can still misread or go beyond its sources, so citations and evaluation matter.

Sources and further reading

Think you understand it now? Prove it.

Solve real Tensor Core math in the TensorCore.si Challenge and earn a certificate with your name.

Take the challenge

Leave a Reply

Your email address will not be published. Required fields are marked *