Evaluate your RAG system

To know whether your RAG system answers well, you measure two things separately: did retrieval fetch the right passages, and did the model answer faithfully from them. You build a small labelled question set, score each stage against it, and re-run those scores on every change. Vibes are not evaluation.

If you are still assembling the pipeline, read RAG fundamentals first. This guide is about proving it works.

Why do you score retrieval and generation separately?

A RAG answer can be wrong for two different reasons, and they need different fixes:

• Retrieval missed. The passage that holds the answer never reached the prompt, so the model had no way to produce it. The fix is in chunking, embeddings, or search.
• Generation drifted. The right passage was in the context, but the model ignored it, contradicted it, or padded the answer with invention. The fix is in the prompt or the grounding instructions.

These look identical from the outside: both produce a confident wrong answer. If you only score the final answer, you cannot tell which half to repair, so you guess. Scoring the two stages on their own turns guessing into a decision.

Step one: build a golden set

You cannot measure anything without a reference. The reference is a golden setA hand-checked set of questions paired with the correct answer and the documents that contain it. It is the ruler you measure every change against.: a list of real questions, the correct answer for each, and the source passage that answer should come from.

Start with thirty to fifty questions. That is enough to catch a regression and small enough to label by hand in an afternoon. Cover the spread of how people ask:

• Plain lookups. "What is the refund window." One passage holds the answer.
• Multi-passage questions. "Compare the starter and pro plans." The answer needs two or three passages.
• Exact-term questions. "What does error E-4021 mean." Tests whether search finds literal strings, not just meaning.
• Unanswerable questions. "What is the CEO's home address." The corpus does not contain this. The correct behaviour is to decline, and you must test for it.

Store the set as plain data so you can version it next to your code. A row per question is enough:

{
  "id": "q-014",
  "question": "How long is the refund window?",
  "expected_answer": "30 days from the delivery date.",
  "source_docs": ["docs/billing/refunds.md"],
  "answerable": true
}

Claude can draft this set for you. Point the Claude Code CLI at your real documents and ask it to generate candidate questions and answers, then you check them. Drafting is fast; the value is in your verification, so do not skip the read-through.Common gotchaA question drafted straight from a passage tends to echo that passage's exact wording, so retrieval finds it easily and your recall looks great while real users phrasing things differently get worse results.During the read-through, reword each question into how a customer would ask, with their own synonyms, so the set tests retrieval instead of rewarding string overlap.

Step two: score retrieval on its own

Run each golden question through retrieval only and compare the passages it returns against the source_docs you labelled. Three numbers tell you most of what you need.

Recall@kRecall at k: the share of relevant passages that appear in the top k results your retriever returns for a query. is the one to watch. If the answer is not in the retrieved context, no amount of prompt tuning will recover it, so a retrieval miss is a hard ceiling on answer quality. Track it first.

Precision@kPrecision at k: of the k passages you returned, the share that were relevant to the question. is the counterweight. You can always raise recall by returning more passages, but each junk passage costs tokens and gives the model more to be distracted by. You want both high, and k set so recall is strong without burying the good passage.

MRRMean Reciprocal Rank: the average of one-over-the-rank of the first correct result across all questions. Rewards putting the right passage early. rewards ordering. A correct passage at position one is worth more than the same passage at position eight, because models attend best to the start and end of a long context. If MRR is low while recall is fine, a rerankerA second-pass model that re-reads the query and each top passage together and re-scores them, pushing the most relevant one to the top. is the usual fix.

How do recall and precision trade off?

The two move against each other as you change k, the number of passages you return. Raise k and recall climbs, because more passages means a higher chance the right one is included. But precision falls, because each extra passage is more likely to be junk. The job is to find the k where recall is already high and precision has not yet collapsed.

A worked example makes the trade concrete. Say a question's answer lives in one passage:

• At k=3, you return three passages, the right one is among them, and two are noise. Recall is high, precision is one in three.
• At k=10, recall is still high but precision drops to one in ten, and the model now reads nine irrelevant passages looking for the one that matters.

Sweep k across your golden set and pick the smallest value where recall plateaus. Returning more after that point buys nothing and costs tokens.

Step three: score the generated answer

Once retrieval is solid, score what the model does with it. The four metrics popularised by the Ragas evaluation framework split cleanly across the two stages:

FaithfulnessThe share of claims in the answer that are directly supported by the retrieved passages. Low faithfulness means the model is inventing on top of good context. is your guard against confident invention. A faithful answer makes no claim the context does not support. When faithfulness drops while context recall stays high, the bug is in generation: the model has the facts and is adding to them anyway. Tighten the grounding instruction.

Answer relevancyA measure of whether the answer is on-topic and addresses what was asked, rather than being correct but evasive or padded. catches the opposite problem: an answer that is technically grounded but dodges the question or pads it with filler.

The four metrics as a diagnosis grid

Read the two retrieval metrics and the two generation metrics together and they point straight at the broken stage:

Always fix retrieval before generation. A generation metric measured on top of bad context is noise: the model cannot be faithful to facts it never received, so the faithfulness number means nothing until recall is solid.

Scoring faithfulness by hand does not scale. The common approach is an LLM-as-judge: you give a model the question, the retrieved context, and the answer, and ask it to rule on whether each claim is supported. Treat the judge as an instrument you also calibrate. Spot-check its rulings against your own on a handful of cases before you trust the aggregate.Common gotchaThe trap is treating the judge's faithfulness score as ground truth and tuning your whole pipeline to lift it, without ever checking whether the judge agrees with a human.An uncalibrated judge has its own blind spots, so you can spend a week chasing a number that does not track real answer quality, then ship a system your users find worse. Hand-label twenty answers, measure how often the judge matches you, and fix the judge prompt until that agreement is high before you trust a single aggregate score.

Running it through the CLI

You do not need a separate evaluation service to start. Drive the loop with the Claude Code CLI: it reads your golden set, runs each question, and writes a scored report you can diff against the last run.

A slash commandA reusable, named instruction set you store in your project so Claude runs the same routine the same way every time you invoke it. keeps the routine identical between runs. Save the instructions once and invoke them by name:

---
description: Score the RAG pipeline against the golden set
---
 
Read eval/golden-set.json. For each question:
1. Retrieve passages and record which source_docs appeared and at what rank.
2. Generate an answer from only those passages.
3. Judge faithfulness and answer relevancy against the context.
 
Compute recall@k, precision@k, MRR, and average faithfulness.
Write results to eval/runs/<date>.json and print a table comparing
this run to the previous file in eval/runs/.

Then each run is one line:

claude /eval-rag

Because the routine lives in a file, every teammate runs the same checks the same way, and the diff between two run files is your regression signal. To wire the same routine into a richer toolchain, see MCP fundamentals for connecting evaluation data sources, and hooks and guardrails to fail a commit when a score drops.

How do you catch RAG regressions automatically?

The point of a saved score is to compare against it. Run the evaluation on a schedule and on every change to retrieval, chunking, the prompt, or the index, then alert when a number falls.

A pre-commit or CI hook can run the suite and block a change that drops faithfulness or recall below a floor you set:

{
  "hooks": {
    "pre-commit": "claude /eval-rag && node eval/check-thresholds.js"
  }
}

The threshold check reads the newest run file and exits non-zero if, for example, recall@5 is under 0.85 or faithfulness is under 0.90. Pick the floors from your own baseline, not from a blog post: run the suite once on the system you have today and set the floor a little under each current number.

What should you track over time?

A single run is a snapshot. The signal is in the trend, because the thing that breaks RAG in production is not a bad launch, it is quiet drift as the corpus changes.

• Retrieval recall@k. The leading indicator. A slow decline means your index has drifted from the live source, or new content is chunked badly. Re-index and re-check.
• Faithfulness. A drop here without a retrieval drop points at a prompt change or a model swap that loosened grounding.
• Unanswerable accuracy. The share of unanswerable questions the system correctly declines. This one degrades silently and is where confident invention hides.
• Latency per stage. Retrieval time and generation time, tracked apart. A reranker that lifts MRR but doubles latency is a trade you want to see, not discover in a complaint.
• Cost per query. Returning more passages raises recall and tokens together. Watch them as a pair so a recall gain does not quietly become a budget problem. See token usage and cost.

A worked loop

Put it together and the workflow is short:

1. Build the golden set, thirty to fifty labelled questions including unanswerable ones.
2. Run retrieval alone. Read recall@k first. If it is low, fix chunking, embeddings, or hybrid search before touching anything else.
3. Once retrieval holds, run the full pipeline and score faithfulness and answer relevancy with an LLM judge you have spot-checked.
4. Save the run, set thresholds a little under today's numbers, and wire the suite into a hook.
5. Re-run on every change and on a schedule. Investigate any number that falls.

Evaluation is not a phase you finish. It is the instrument that tells you whether each change helped, and the only honest answer to "is the RAG system good." Next, read RAG for coding agents, where the standard recipe needs adjusting.