Now you have something you can test reliably. Part 2 is about deciding what to test and building the checks. It covers four stages: building a dataset, finding out how the system fails, writing an evaluator for each failure, and measuring whether your AI judges can be trusted.

Stage 2: Dataset

Goal

Build a set of questions with known correct answers and the evidence for them, covering the kinds of questions your system will face.

Why it matters

Every number you report later is a statement about this dataset. If the questions are all easy, your scores will look great and mean little. If the expected answers are wrong, you'll fail correct answers and pass wrong ones. And if you tune the system on the same questions you report, your scores will flatter you. A dataset is the test suite of an evaluation; it deserves the same care.

Key terms

An evaluation dataset is the set of test cases you evaluate against. You'll also hear it called a golden dataset, and its items goldens: the industry's name for a carefully checked set of cases with trusted expected outputs. The trusted outputs are the ground truth.

For a RAG system, each case has three parts: the question, the expected answer (the reference answer from Part 1), and the evidence. Evidence is the exact text in the documents that supports the answer, usually stored as evidence quotes: verbatim spans copied from a document, with the document's name.

Dimensions are the ways questions vary, decided up front so the dataset covers them on purpose. Typical dimensions for RAG:

• Question type: a single fact, a procedure, a cause, a comparison, or a multi-document question that needs facts from two or more documents.
• Complexity: one evidence span, several spans, or an inference across them.
• Phrasing: close to the document's wording, or paraphrased in different words.
• Evidence location: where the answer sits, such as a table, the narrative or the conclusions.
• Chunk boundary: whether the evidence lies inside one chunk or crosses the cut between two.

Some questions should be unanswerable, as Part 1 described. There are three useful kinds:

• Missing detail: about a topic the corpus covers, asking for a detail it never states.
• Out of corpus: about something the corpus doesn't cover at all.
• False premise: built on an assumption that contradicts the documents.

The research community added unanswerable questions to benchmarks for exactly this reason: when a test set only contains answerable questions, systems that guess on everything still score well, so the test never rewards knowing when not to answer [1].

A closed-book baseline runs the questions through the model with no documents at all, so it can only answer from its parametric knowledge (Part 1). Research on this setup shows large models can answer many factual questions from memory alone [2]. For your evaluation, the closed-book score tells you how much of a correct RAG answer could have come from memory rather than retrieval.

Finally, the dataset is split. The dev split (short for development) is the part you look at, learn from and tune against. The held-out split (or test split) is kept aside and used only for final measurements. Tuning prompts or settings while watching held-out scores is overfitting to the eval set: the system gets better at your test questions, not at the job.

How it's done

1. Define the dimensions and targets. Write down each dimension, its values, and roughly what share of the dataset each should have. As a rule of thumb, make around 15–20% of questions unanswerable: enough to measure abstention properly, but not so many that the dataset stops looking like real traffic. Also make sure some evidence crosses a chunk boundary.

2. Write or generate candidate questions. You can write them by hand, or use an LLM to draft candidates from the documents against a plan that fills each dimension. Generated candidates save time, but they are drafts: they need checking like any test data you didn't write yourself.

3. Check them automatically. Rules catch a lot before a person reads anything:

• every evidence quote must appear verbatim in the document;
• near-duplicate questions are removed;
• questions that copy most of their wording from the evidence are flagged as too easy, because keyword search would find them;
• where the evidence sits, and whether it crosses a chunk boundary, is measured from the real text, not assumed.

4. Review every item by hand. Read each question with its evidence in context, and approve, edit or reject it. Check the expected answer is right and complete, and that an unanswerable question really is unanswerable.

5. Split into dev and held-out sets. Split so that each question type appears in both (a stratified split), with a fixed random seed so the split can be reproduced. Record a hash of each file.

6. Run a closed-book baseline with the model and no documents, and check how many questions it gets right from memory.

Here's an example of a single case, with an invented question and document:

{
  "id": "q-042",
  "split": "dev",
  "question": "What speed was the ferry making when it hit the pier, and what time did the impact happen?",
  "answerable": true,
  "expected_answer": "About 6 knots, at 14:32.",
  "evidence": [
    {"doc_id": "report-17.pdf", "quote": "At 14:32 the ferry struck the north pier while making about 6 knots."}
  ],
  "dimensions": {"question_type": "factual", "complexity": "single_span", "phrasing": "paraphrased"}
}

And a false-premise question built on the same invented report: "Why did the ferry's captain ignore the harbour pilot's warning?", when the report says no pilot was aboard. The right answer points out the false assumption instead of inventing a reason.

✅ Best practices

• Build from explicit dimensions. Questions you write freely cluster around what's easy to ask; a plan makes coverage deliberate and visible.
• Paraphrase away from the source wording. Real users don't quote documents; questions that copy the text test keyword matching, not retrieval of meaning.
• Include unanswerable and multi-document questions. Without them you can't measure abstention [1], and you won't find out whether the system can combine evidence.
• Label evidence as verbatim quotes, not chunk IDs. Chunks change whenever you change chunking; a quote stays valid, so your labels survive configuration changes.
• Review every item by hand. A wrong expected answer quietly corrupts every score built on it.
• Keep a held-out split you never tune on. It's the only honest estimate of how the system will do on questions you haven't optimised for.
• Run a closed-book baseline. If the model can answer from memory, a correct RAG answer doesn't prove retrieval worked [2].
• Plan for judge validation now. Stage 5 needs enough failing examples for each failure type; a rare failure type needs more questions of that kind.

⚠️ Common mistakes

• Only answerable, single-fact questions, so scores look high and abstention is never tested.
• Trusting generated questions and expected answers without reading them.
• One split for everything, so tuning and reporting use the same questions.
• Evidence stored as chunk IDs, which break the first time chunking changes.

Stage 3: Baseline runs and error analysis

Goal

Find out how the system actually fails, before choosing a single metric.

Why it matters

It's tempting to open an evaluation framework, pick its built-in metrics and start measuring. The trouble is that generic metrics measure generic problems. Your system fails in its own specific ways, and you only find out which by looking. Experienced practitioners put this first: read real outputs, then build evals around the failures you see [3]. Measuring before looking gives you precise numbers about the wrong things.

Key terms

A baseline is the configuration you treat as the reference: your best current setup, which later changes are compared against.

A weak configuration is a deliberately worse variant, such as fewer retrieved passages or a prompt that doesn't ask for grounding. Running a few weak configurations alongside the baseline gives you a wider range of failures to learn from.

Error analysis is the process of reading traces, describing what went wrong, and organising those descriptions into categories. It borrows two techniques from qualitative research [4]:

• Open coding: reading each trace and writing a short note in your own words about what went wrong, without a predefined list of categories.
• Axial coding: grouping those notes into categories, merging similar ones and naming each group.

Each resulting category is a failure mode, and the full set, with definitions and examples, is a failure taxonomy.

When several things go wrong in one trace, note only the first upstream failure: the earliest point in the pipeline where it broke. If retrieval missed the evidence, the wrong answer that follows is a consequence, not a separate failure. Recording only the first cause keeps you from double-counting one problem as two.

How it's done

1. Run the baseline and a couple of weak configurations on the dev split. Never the held-out split: you'll be looking closely at every trace, and that's tuning-by-eye.

2. Review the traces blind. Shuffle them and hide which configuration produced each one, so your expectations ("this is the weak config, it'll be bad") don't colour your notes. Show everything needed to judge each one: the question, expected answer and evidence, the retrieved passages (ideally with the evidence highlighted wherever it appears), and the answer.

3. Write one short note per trace about the first upstream failure, or mark it as fine. For example, three invented notes:

• "The passage with the impact time wasn't retrieved; the answer says the documents don't say."
• "The right passage was retrieved, but the answer gives 16 knots instead of 6."
• "The answer explains why the captain ignored the pilot, but no pilot was aboard."

4. Group the notes into failure modes. The first note becomes "answer not retrieved", the second "right passage, wrong reading", the third "trick question not caught". Name each mode, define it in one sentence, and count how many traces fall into it.

As a tester, you'll recognise this as exploratory testing plus defect triage. The difference is where the categories come from. In triage you usually file defects into categories that already exist; here you build the categories from what you find.

✅ Best practices

• Do error analysis before choosing metrics. Evals built from observed failures measure what actually goes wrong in your system [3].
• Attribute each failure to its first upstream cause. It separates retrieval problems from generation problems, which need different fixes.
• Have a person do the reading. Open coding is where you decide what counts as a failure. If a model does it, its blind spots become your taxonomy, and the judges you build on that taxonomy will share them.
• Hide the configuration while reviewing. Knowing which config produced a trace biases what you see in it.
• Write notes in your own words first, group later. Starting from a fixed list of categories makes you see only those categories.
• Expect your criteria to sharpen as you read. Researchers call this criteria drift: grading outputs changes what you think the criteria should be [5]. Let the taxonomy settle before you build evaluators on it.

⚠️ Common mistakes

• Starting from a framework's list of metrics instead of from your own traces.
• Blaming generation for a wrong answer when retrieval never found the evidence.
• Doing error analysis on the held-out split.
• Reading only the failures you expected, or too few traces to see the patterns.

Stage 4: Evaluators

Goal

Build one automated check for each failure mode that actually occurs in your taxonomy.

Why it matters

Error analysis tells you what goes wrong today. Evaluators let you measure it every time the system changes, without reading hundreds of traces by hand. Building them from the taxonomy keeps each check tied to a real problem, and keeps the set small: no check for a failure you've never seen, no failure without a check.

Key terms

An evaluator (or metric) is a check that scores one aspect of a trace.

A deterministic evaluator follows a fixed rule and always gives the same result for the same input. Exact-match checks and retrieval metrics are deterministic. An evaluator that calls an LLM is not.

A reference-based evaluator compares the output against something known in advance, such as the expected answer or the evidence. A reference-free evaluator judges the output on its own terms, for example checking that every claim is supported by the retrieved passages, which needs no expected answer.

Retrieval metrics measure whether retrieval found the evidence, using the evidence quotes from your dataset. The three most common:

• Hit rate: the share of questions where at least one of the top k retrieved passages contains evidence.
• Recall@k: for one question, the share of its evidence found within the top k retrieved passages, averaged over questions.
• MRR (mean reciprocal rank): for each question, take 1 divided by the rank of the first passage that contains evidence (1 if it's first, ½ if second, ⅓ if third, 0 if none), then average over questions. It rewards finding the evidence early.

A quick worked example, with invented numbers. A question has two evidence quotes. Four passages come back. The first is unrelated, the second contains quote A, the third is unrelated, and the fourth contains quote B. That question scores a hit, recall@4 of 2/2, and a reciprocal rank of ½.

You can measure these at document level (did any passage come from the right document?) or at passage level (did a passage contain the actual evidence?). Document level is easier to pass and tells you less: finding the right report isn't the same as finding the paragraph with the answer. Match passages to evidence by their text, since chunk IDs change whenever chunking changes.

For everything a rule can't decide, you use LLM-as-a-judge, introduced in Part 1: a separate model that grades an output against your instructions. The instructions are the judge prompt. The criteria inside it, with what counts as passing and failing, are the rubric. A binary (pass/fail) judge answers one yes-or-no question. A scale judge gives a score, such as 1 to 5. Binary judges are easier to make consistent and to check against human labels, and nobody has to decide what separates a 3 from a 4 [6].

Good judge prompts include few-shot examples: a handful of worked examples inside the prompt showing the judge how to decide, without retraining the model [7]. A useful set is one clear pass, one clear fail and one borderline case with its decision explained.

Common judges for RAG, each matched to failure modes you might find:

• Groundedness (also called faithfulness): is every claim in the answer supported by the retrieved passages? Catches unsupported claims, guesses and facts borrowed from the wrong document.
• Correctness: does the answer give the facts in the expected answer? Catches misreadings and wrong answers.
• Abstention: on an unanswerable question, does the answer decline or point out the false premise?
• Unnecessary refusal: on an answerable question, does the answer hold back something the passages contain?

How it's done

1. Use rules wherever a rule can decide. If the system returns a fixed refusal text when nothing is retrieved, an exact-match check identifies that case with no model call. Retrieval metrics need no judge either.

2. Write one binary judge per remaining failure mode. Give it one question to answer, explicit Pass and Fail definitions, three labelled examples, and a fixed output format. A shortened example:

You are grading one answer from a question-answering system over accident reports.
Judge one thing only: is every factual claim in the answer supported by the passages?

Fail if the answer states a number, name, time or cause the passages do not support,
uses facts from a passage about a different accident, or offers a guess ("likely",
"probably") as an answer. Pass if every claim is in the passages or follows directly
from them. Saying the passages do not contain the answer is a Pass.

[three short worked examples: clear pass, clear fail, borderline]

Reply with JSON only: {"result": "Pass" or "Fail", "reason": "<one sentence>"}

And a verdict it might return:

{"result": "Fail", "reason": "The passages give 6 knots; the answer says 16 knots."}

3. Show each judge only what it needs. A groundedness judge needs the passages and the answer, not the expected answer, or it may start grading correctness instead. A correctness judge needs the expected answer, not the passages.

4. Choose the judge model. Use a model from a different family than the system under test, because, as Part 1 mentioned, judges tend to favour outputs like their own. Pin its version like any other component.

5. Control cost. Judges cost a model call each. Cache verdicts keyed on the judge model, the prompt and the inputs, so unchanged cases are never graded twice. Estimate the cost before each run, and set a cap that stops a run before it overspends.

If you think of a judge as a test oracle, here's where the comparison breaks: you trust an oracle, but a judge is a model with its own error rate. Until you've measured that error rate, a judge's pass rate is an opinion, not a measurement. That's the next stage.

✅ Best practices

• Prefer deterministic checks. They're free, exact and repeatable; use a judge only where a rule can't decide.
• Use binary judges, one failure mode each. One clear question per judge is easier to make consistent and to validate than a single "quality" score [6].
• Give explicit definitions and labelled examples. The judge should never have to guess where the line is [7].
• Pick a judge from a different model family. It reduces the risk of the judge favouring answers in its own style [8].
• Measure retrieval at passage level against evidence text. Document-level metrics can look fine while the evidence itself is rarely retrieved.
• Cache verdicts and cap cost. A re-run should never pay twice for the same judgement.

⚠️ Common mistakes

• Scoring answers 1–10, so nobody can say what a 6 means or whether it changed.
• One judge for "overall quality", which can't tell you what went wrong.
• Showing a judge the expected answer when it's meant to check grounding.
• Using built-in metrics that don't match any failure mode in your taxonomy.

Stage 5: Judge validation

Goal

Measure how often each judge agrees with a careful human, before you trust its numbers.

Why it matters

A judge that's wrong 30% of the time can make a broken system look fine, or a good one look broken. Researchers studying LLM evaluators put it plainly: LLM-based graders inherit the problems of the LLMs they evaluate, so they need checking against human judgement [5]. Validation is what turns "the judge says 85%" into a number with known error bars.

Key terms

Human labels are Pass or Fail verdicts that a person assigns to real traces, for the same question each judge answers. They're the answer key for testing the judge, and judge validation is comparing the judge's verdicts with them.

Treat each judge as a classifier with two possible errors: it can pass an answer a human failed (a missed failure), or fail an answer a human passed (a false alarm).

From that table come two rates. Conventions vary, so here's how this series uses them:

• True positive rate (TPR): of the answers a human passed, the share the judge also passed. Low TPR means false alarms.
• True negative rate (TNR): of the answers a human failed, the share the judge also failed. Low TNR means missed failures.

Why not just report accuracy, the share of all verdicts that agree? Because it hides the error you care about when failures are rare. Imagine 100 answers, 90 of them good. A judge that passes everything scores 90% accuracy and catches none of the 10 failures: TPR 100%, TNR 0%. Reporting both rates exposes this immediately [6].

Judges also have known biases. Studies of LLM judges have found position bias (favouring the answer shown first when comparing two), verbosity bias (favouring longer answers) and self-enhancement bias (favouring their own outputs) [9]. Binary questions about one answer avoid the first, explicit rubrics reduce the second, and a judge from a different family reduces the third.

To validate without fooling yourself, split the labels like the dataset. Judge-dev labels are the ones you look at while improving the judge prompt. Judge-test labels are used once, at the end, for the numbers you report.

How it's done

1. Collect traces with enough passes and fails for each judge. Use dev traces from the baseline and from weak configurations, which supply failures. Aim for roughly 50 passes and 50 fails per judge where you can. With far fewer, the uncertainty around TPR and TNR gets too wide to be useful (Part 3 shows how to put numbers on that uncertainty). If you have real user traffic, sample traces from it too: it contains failures your dataset didn't anticipate.

2. Label them blind. A person marks Pass or Fail for each judge question, without seeing the judge's verdicts and without knowing which configuration produced the trace. Leave out any trace used as a few-shot example in a judge's prompt.

3. Split the labels by question into judge-dev and judge-test, so the same question never appears on both sides.

4. Improve on judge-dev. Run the judges, read the disagreements, and fix unclear wording in the prompts. Stop when the remaining disagreements are genuinely borderline, not when the numbers look good.

5. Measure once on judge-test, and report TPR and TNR for each judge, with how many passes and fails each rate is based on.

6. Re-validate whenever the judge model or its prompt changes. The numbers belong to that exact model and prompt.

✅ Best practices

• Validate every judge against human labels before using its scores. An unvalidated judge's numbers have unknown error [5].
• Report TPR and TNR, not accuracy alone. Accuracy can look high while a judge misses every failure [6].
• Keep judge-dev and judge-test labels separate. Tuning a prompt on the labels you report makes the result look better than it is.
• Get enough failures per judge. If a failure type is rare, add questions or weak configurations that provoke it, or report the judge as not validated.
• Re-validate after any change to the judge model or prompt.

⚠️ Common mistakes

• Testing a judge on the same traces used as its few-shot examples.
• Improving the prompt while watching judge-test results.
• Labels made by another AI instead of a person, which measures how much two models agree, not whether the judge matches human judgement.
• Reporting a judge as validated when it was tested on only a handful of failures.

Recap and what's next

You built a dataset from explicit dimensions, with verbatim evidence, unanswerable and multi-document questions, a held-out split, and a closed-book baseline to check for memorisation. You read real traces before measuring anything, noted the first upstream failure in each, and grouped the notes into a failure taxonomy.

From that taxonomy you built evaluators: rules and retrieval metrics wherever possible, and binary judges with explicit rubrics for the rest. Then you measured each judge against human labels, using TPR and TNR on a held-out set of labels.

You now have checks you can trust, with known error rates. But one question remains: when a score changes, is it a real change or noise? And would your evals actually catch the system getting worse? Part 3: Proving Your Evals Work covers statistics for small samples, breaking the system on purpose to test the evals, a quality gate for CI, and making every result reproducible.

How do you build test data and decide what to measure for your AI systems today? Tell me in the comments.

References

1. Rajpurkar, P. et al. (2018). Know What You Don't Know: Unanswerable Questions for SQuAD.
2. Roberts, A. et al. (2020). How Much Knowledge Can You Pack Into the Parameters of a Language Model?
3. Husain, H. (2024). Your AI Product Needs Evals.
4. Strauss, A. and Corbin, J. (1990). Basics of Qualitative Research: Grounded Theory Procedures and Techniques. Sage.
5. Shankar, S. et al. (2024). Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences.
6. Husain, H. (2024). Using LLM-as-a-Judge For Evaluation: A Complete Guide.
7. Brown, T. et al. (2020). Language Models are Few-Shot Learners.
8. Panickssery, A. et al. (2024). LLM Evaluators Recognize and Favor Their Own Generations.
9. Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.

This series is a ground-up guide to evaluating that kind of system, called a RAG system, written for testers. It assumes you know testing well and starts from zero on machine learning. Over three parts you'll learn what to measure, how to build the checks, and how to prove the checks themselves work.

Part 1 covers the foundations: what a RAG system is, where it fails, which settings you'll vary, which kinds of tools you need, and the first two stages of an evaluation: getting to know the system, and making every test run reproducible.

Why testers are well placed to evaluate AI

Most of what you already do carries over. You design test cases around risks. You write down expected results before you run anything. You keep a regression suite so a change that breaks something old gets caught. You triage defects to find where they really started. You put quality gates in CI so bad builds don't ship. Evaluating an AI system needs every one of these habits.

What changes is the thing you're testing. Three differences matter most.

The output isn't deterministic. A large language model (LLM) is the kind of AI model that reads text and writes text, like the models behind ChatGPT or Claude. It writes its answer one small piece at a time, picking each piece from many possible continuations. Ask the same question twice and you can get different wording, different detail, or now and then a different answer. That's non-determinism. Settings can reduce it, but usually can't remove it: even with the randomness turned all the way down, the servers that run models can return different outputs for the same input, because of how they batch requests together [1]. You've dealt with flaky tests before; here, some flakiness comes with the territory. A single passing run tells you less than you're used to.

There's often no single correct output. "Paris is the capital of France" and "France's capital is Paris" are both right. So instead of an exact expected result, you usually have a reference answer: a correct answer you compare against, allowing for different wording.

The automated checker can be wrong. Comparing free-text answers by hand doesn't scale, so teams use another LLM to grade answers against rules. This is LLM-as-a-judge, and a model used this way is a judge. Think of a judge as a test oracle, the thing that decides whether a test passed, and the analogy works up to a point. You normally trust a test oracle. A judge is a model with its own error rate, so you have to measure how often it's right before its results mean anything. Part 2 shows how.

None of this makes AI untestable. You test it with the same discipline, plus some statistics and a way of checking the checker. A structured test of an AI system's quality is called an eval (short for evaluation), and that's what this series builds.

What RAG is

An LLM learns from a huge amount of text during training, then stops learning. It doesn't know your company's internal documents, or anything that happened after training. Ask it anyway and it may answer confidently from general knowledge, or simply make something up.

Retrieval-augmented generation (RAG) fixes that. Before the model answers, the system looks up relevant text in a set of documents and hands it to the model along with the question. The model is told to answer from what it was given. Lewis and colleagues introduced the approach in a 2020 paper [2], and it's now one of the most common ways companies put LLMs to work on their own data.

A RAG system has two halves:

• Retrieval finds the pieces of text most relevant to a question.
• Generation is the LLM writing an answer from the question plus those pieces.

The full set of documents the system can search is the corpus. One file in it is a document, and a piece of a document that retrieval returns is a passage. The passages handed to the model are its context.

One request flows like this:

1. A user asks a question.
2. The system converts the question into a form it can compare with the stored documents (more on that in the next section).
3. It searches the corpus and keeps the few passages that match best.
4. It builds a prompt: instructions, the retrieved passages, and the question.
5. The LLM reads the prompt and writes an answer.

Each step can fail in its own way. If you remember one idea from this article, make it this one.

Retrieval can miss: the passage with the answer is in the corpus, but it isn't among the passages returned. Or retrieval finds the right document but the wrong part of it.

Generation can fail even when retrieval did its job:

• Hallucination: the model states something that its context, or reality, doesn't support. The term is popular but loosely defined. For RAG, grounding is the more precise idea.
• Not grounded: a grounded answer is one where every claim is supported by the retrieved passages. An answer that adds facts from the model's general knowledge, or from a passage about something else, isn't grounded, even if the extra facts happen to be true.
• Misreading: the right passage is there, but the model takes the wrong number or credits an action to the wrong person.
• Refusal problems: a refusal is the system declining to answer, and sometimes that's correct. An unanswerable question is one the documents can't answer: the detail isn't there, the topic isn't covered, or the question rests on a false assumption. For those, the right behaviour is abstention: declining, or pointing out the false assumption. The opposite failure is refusing when the answer was in the passages all along.

Why evaluate the two halves separately? Because a wrong answer alone doesn't tell you what to fix. If retrieval never found the right passage, rewriting the model's instructions won't help. You know this principle already: a failing end-to-end test needs narrower tests behind it before you can locate the defect.

The building blocks you'll vary and test

A RAG system has a lot of settings, and each one changes its behaviour. These are the variables you'll record, pin, and sometimes change on purpose to test your evals. They fall into four groups.

Splitting documents into chunks

Documents are usually too long to search or to hand to a model whole, so the system cuts them into smaller pieces called chunks. That process is chunking. The chunk size is how long each piece is, often counted in characters or tokens. A token is the unit a model reads, roughly a word or part of a word.

Cut a sentence in half at a chunk boundary and the two halves may never be retrieved together. Chunk overlap repeats a little text from the end of each chunk at the start of the next, which lowers that risk.

Chunk size is a trade-off. Small chunks are precise but can lose the surrounding context. Large chunks carry more context, and more noise.

Finding the right passages

To search by meaning rather than exact words, the system turns each chunk into an embedding: a long list of numbers that represents what the text is about. Texts with similar meaning get similar numbers. The model that does this is the embedding model, or embedder. It's a separate model from the LLM that writes answers, usually much smaller, and swapping it changes what retrieval finds.

Embeddings live in a vector database, a store built to find the embeddings closest to a given one. At question time the question is embedded the same way, and a similarity search returns the closest chunks. Each result comes with a similarity score saying how close it is. A common measure is cosine similarity, where higher means more similar. Some systems report a distance instead, where lower means closer, so check which one yours uses before you set a threshold.

Two settings decide what's kept:

• Top-N (also written top-k) is how many of the best-scoring chunks go to the model, for example the top 4.
• The similarity threshold is a minimum score. Chunks below it are dropped, even if they're in the top N.

Set the threshold too strict and the model gets nothing. Too loose, and irrelevant text gets in.

Producing the answer

The text sent to the LLM is the prompt. Part of it is the system prompt: standing instructions from whoever configured the system, such as "Answer only from the provided documents. If they don't contain the answer, say so." The system prompt has a big effect on grounding and refusals.

Three model-level settings matter:

• Context window: the most text, in tokens, the model can read at once. The system prompt, passages and question all have to fit.
• Temperature: how much randomness goes into picking each next token. At 0 the model always takes the most likely continuation, which gives the most repeatable output. Higher values give more varied output. As mentioned earlier, 0 reduces variation but doesn't guarantee identical results.
• The model itself: a local model runs on your own hardware; a hosted model runs on a provider's servers and you call it over an API. Some models are reasoning (thinking) models: they write out intermediate reasoning before the answer. That's usually slower, and it changes what you need to record.

How the system behaves

Many RAG applications bundle a document set and its settings into a separate unit. AnythingLLM calls it a workspace. One workspace per configuration lets you compare configurations without them interfering with each other.

Most also let you pick a mode. In a document-only mode (AnythingLLM calls it query mode), the system answers only from retrieved context and returns a fixed refusal when nothing relevant turns up. In a conversational mode (chat mode), the model answers even with no relevant context, falling back on its general knowledge. If you're evaluating a system that's supposed to answer from documents, document-only mode matches the job. As a bonus, the fixed refusal text is something you can check exactly.

Here's an example configuration with the kind of settings you'll pin. The names and values are illustrative, not recommendations:

# Example only: one configuration of a RAG system under test
llm:
  model: example-model-8b
  temperature: 0
  context_window: 8192
embedder:
  model: example-embedder
chunking:
  size: 1000      # characters
  overlap: 20
retrieval:
  top_n: 4
  similarity_threshold: 0.25
generation:
  mode: query
  system_prompt: >-
    Answer only from the provided context. If the context does not contain
    the answer, say that the documents do not contain it.
  refusal_response: NO_RELEVANT_CONTEXT

Every line in that file can change an answer. That's why the coming stages treat configuration as part of the test, not as background.

The kinds of tools you'll need

You don't need a particular product to follow this series. You need four kinds of tools:

Running models locally has real advantages for evaluation. Calls cost nothing, your documents stay on your machine, and you can pin the exact model version so it doesn't change under you. The catch: you need hardware that runs the model at a usable speed, and long runs keep that hardware busy and hot.

An evaluation framework saves you writing plumbing: it organises test cases, runs metrics and reports results. What it can't do is tell you which failures your system actually has, or whether its built-in judges are any good on your data. Those calls are yours, and most of this series is about making them well.

Why should the judge come from a different model family? Research has found that LLM judges can recognise their own outputs and tend to rate them more favourably [5], so a model grading its own family's answers can be biased. Part 2 comes back to this.

My view: the tools matter much less than the method. Everything in this series carries over to a different stack.

The road ahead: ten stages

The rest of the series follows an evaluation from first contact with a system to an automated quality gate. Each stage builds on the one before.

Each stage section has the same layout: the goal, why it matters, the key terms, how it's done, best practices and common mistakes.

Stage 0: Spike

Goal

Find out how the system under test really behaves before you design any evaluation around it.

Why it matters

Every eval rests on assumptions:

• that a setting does what its name says;
• that two test cases don't affect each other;
• that you can see what the model was given.

If one of these is false, your results will be wrong in ways that are hard to spot later. Finding out now is cheap. Finding out after weeks of evaluation costs you the weeks.

Key terms

A spike is a short, time-boxed investigation that answers a technical question before you commit to a design. The term comes from Extreme Programming, so you may know it from agile teams.

The system under test is the RAG application in one specific configuration, treated as a black box behind its interface, the same way you'd treat any service you test.

Test isolation means each test case runs without being affected by any other. With RAG systems this is less obvious than it sounds. Many applications keep chat history and feed earlier messages back into later ones. Some go further and reuse passages from earlier turns when retrieval comes up short, so the model sees text the current question never retrieved.

How it's done

List the questions your evaluation depends on, then test each one against the real system and keep the evidence: requests, responses, screenshots. Typical questions:

1. Can you drive the system entirely through an API, without anyone clicking through a UI?
2. Which settings can you set and read back, and do changes actually take effect?
3. Does each response tell you which passages the model saw, and their scores?
4. If you send the same question twice at temperature 0, do you get the same answer?
5. What happens when nothing relevant is retrieved: a fixed refusal, or a guess?
6. Does conversation history carry over between questions, and can you start a fresh session for each one?
7. Are there settings you can't control, such as a random seed, that will limit repeatability?

Here's the kind of experiment that pays off. Imagine you load a test system with two short, fictional documents, so any correct answer has to come from retrieval rather than the model's training. You ask a question whose answer exists in only one of them, with the similarity threshold set so high that nothing gets retrieved. In document-only mode you might get the fixed refusal text straight back. In conversational mode, the same setup might produce a confident, invented answer. One small experiment, and you know which mode to evaluate and what a refusal looks like in your traces.

✅ Best practices

• Verify behaviour before designing evals. Every later stage assumes the system behaves as documented. Checking costs hours; finding a wrong assumption after weeks of evaluation costs the weeks.
• Use fictional test documents for the spike. The model can't have seen them in training, so a correct answer proves retrieval worked.
• Keep evidence for each finding. Saved requests and responses let you re-check a conclusion when the system is upgraded.
• Score retrieval on what the model actually received. Use the system's own record of the passages it passed to the model, not a separate search, so retrieval and generation are judged on the same input.

⚠️ Common mistakes

• Trusting setting names or documentation without testing them.
• Running test cases in one long conversation, so earlier questions influence later answers.
• Assuming temperature 0 means identical outputs, and treating a single run as the final word.
• Testing only the happy path, and never checking what happens when retrieval comes back empty.

Stage 1: Foundation

Goal

Make every evaluation run reproducible and every test case independent, so that a difference in results means a difference in the system, not in how the run was set up.

Why it matters

You'll compare runs constantly: before and after a prompt change, one model against another, this week's build against last month's. A comparison only means something if everything you didn't intend to change stayed the same. Without a foundation, a score drop could come from many sources:

• a model that was quietly updated;
• a setting someone changed in the UI;
• a test case that inherited history from the one before it.

Key terms

Configuration as code means the system's settings live in version-controlled files and are applied automatically, never edited by hand. It's the same idea as infrastructure as code. Provisioning is the step that applies a configuration: creating the workspace, loading the documents and setting every value from the file. Provisioning should be idempotent: running it twice leaves the same state as running it once.

A test case (usually just case) is one question with its expected outcome. A run is one execution of a configuration over a set of cases.

For each case, a run records a trace: everything you need to understand what happened. That's the question, the passages retrieved with their scores, the answer, the model that answered, the latency, and any error. Traces are what you'll read in Part 2 to find out how the system fails.

Each run also writes a manifest: a record of exactly what ran. It holds:

• the code version;
• the configuration and its hash (a short fingerprint that changes if any setting changes);
• the versions of every component;
• hashes of the dataset and documents.

Pinning means referring to an exact, unchangeable version of each component instead of a label that can move. Container images and many model registries give each version a digest, a content hash that identifies one exact version. A tag like latest can point to something different tomorrow [3].

The corpus needs care too. What an LLM knows from training is its parametric knowledge. If the model already knows the answers, a correct answer doesn't prove retrieval worked; it might have come from memory. When test material ends up in a model's training data, that's contamination (or leakage), and it quietly inflates scores [4]. Part 2 shows how to measure it directly with a closed-book baseline.

How it's done

1. Write the configuration as a file, like the example earlier, and provision the system from it. Each variant for an experiment is a new file that changes one setting, not a hand edit.

2. Run every case in a fresh session, so no history carries over. If the system offers threads or sessions, create a new one per case.

3. Record a trace per case and a manifest per run.

An example trace record:

{
  "run_id": "2026-01-15T10:00:00Z-baseline",
  "case_id": "q-017",
  "question": "Example question about a document in the corpus?",
  "retrieved": [
    {"doc_id": "report-12.pdf", "score": 0.81, "text": "Example passage text..."},
    {"doc_id": "report-12.pdf", "score": 0.77, "text": "Another passage..."}
  ],
  "answer": "Example answer text.",
  "model": "example-model-8b",
  "latency_s": 1.4,
  "error": null
}

And an example manifest:

{
  "run_id": "2026-01-15T10:00:00Z-baseline",
  "git_commit": "a1b2c3d",
  "config_hash": "9f8e7d...",
  "app_image": "example/rag-app@sha256:5461...",
  "llm": {"name": "example-model-8b", "digest": "0aaa..."},
  "embedder": {"name": "example-embedder", "digest": "0a10..."},
  "dataset_sha256": "6dec...",
  "corpus_sha256": "bcb8..."
}

4. Pin every version. That covers:

• the application image, by digest;
• the models, by digest;
• your own dependencies, through a lockfile (a file that records the exact version of every library);
• for a hosted model, a dated model version rather than an alias that the provider can repoint.

Check the pins at the start of each run, and stop if anything doesn't match.

5. Choose the corpus deliberately. Look for documents the model is unlikely to have memorised, such as material published after the model's training data was collected. They should be deep enough to support questions that need several passages or several documents. And they should be licensed so you can use them for testing (and share them, if you plan to).

✅ Best practices

• Keep configuration as code. A setting that isn't in version control can't be reproduced or reviewed. Same reasoning as infrastructure as code.
• Isolate every test case. Shared history changes what the model sees, so results stop being independent and traces stop being accurate (see Stage 0).
• Pin by digest, not by tag. A tag can be moved to a new version; a digest always means the same bytes [3].
• Write a manifest for every run. Any number you report later should trace back to the exact system that produced it.
• Choose a corpus that limits memorisation, and check the licence. Otherwise correct answers may reflect training data rather than retrieval [4].

⚠️ Common mistakes

• Changing settings by hand in a UI between runs, then not being able to say what changed.
• Using a latest tag for a model or application, so the system changes under you mid-project.
• Saving only the final answer, so you can't tell later whether a failure came from retrieval or generation.
• Testing on famous public documents (well-known articles, popular books) that the model has probably seen in training.

Recap and what's next

A RAG system retrieves passages from a corpus and writes an answer from them, and each half fails in its own way. Retrieval can miss the passage; generation can stray from it, misread it, or get refusals wrong.

Evaluating it takes the testing habits you already have, with three adjustments:

• outputs vary between runs;
• correctness is judged against a reference, not matched exactly;
• an automated judge has to be measured before you trust it.

Before writing a single metric, you spiked the system to confirm how it really behaves. Then you built a foundation where every setting is code, every case runs in isolation, every version is pinned, and every run leaves a trace and a manifest.

In Part 2: Building the Evals, you'll build a test dataset with known answers and evidence, read real failures to decide what to measure, write deterministic checks and AI judges for those failures, and measure how often each judge is right.

If you already test a RAG system, how do you handle these first two stages? Tell me in the comments.

References

1. He, H. and Thinking Machines Lab (2025). Defeating Nondeterminism in LLM Inference.
2. Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.
3. Docker documentation. Pull an image by digest (immutable identifier).
4. Sainz, O. et al. (2023). NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark.
5. Panickssery, A. et al. (2024). LLM Evaluators Recognize and Favor Their Own Generations.