The Tester's Guide to RAG Evaluation, Part 1: Foundations

If you test software, you already ask the question that matters most in AI evaluation: how would I know this is wrong? Now ask it of a system that answers questions from a company's documents. The same question can get two different answers. Both can sound right. And the tool you'd reach for to check them might be another AI, which can be wrong too.

This series is a ground-up guide to evaluating that kind of system, called a RAG system, written for testers. It assumes you know testing well and starts from zero on machine learning. Over three parts you'll learn what to measure, how to build the checks, and how to prove the checks themselves work.

Part 1 covers the foundations: what a RAG system is, where it fails, which settings you'll vary, which kinds of tools you need, and the first two stages of an evaluation: getting to know the system, and making every test run reproducible.

Why testers are well placed to evaluate AI

Most of what you already do carries over. You design test cases around risks. You write down expected results before you run anything. You keep a regression suite so a change that breaks something old gets caught. You triage defects to find where they really started. You put quality gates in CI so bad builds don't ship. Evaluating an AI system needs every one of these habits.

What changes is the thing you're testing. Three differences matter most.

The output isn't deterministic. A large language model (LLM) is the kind of AI model that reads text and writes text, like the models behind ChatGPT or Claude. It writes its answer one small piece at a time, picking each piece from many possible continuations. Ask the same question twice and you can get different wording, different detail, or now and then a different answer. That's non-determinism. Settings can reduce it, but usually can't remove it: even with the randomness turned all the way down, the servers that run models can return different outputs for the same input, because of how they batch requests together [1]. You've dealt with flaky tests before; here, some flakiness comes with the territory. A single passing run tells you less than you're used to.

There's often no single correct output. "Paris is the capital of France" and "France's capital is Paris" are both right. So instead of an exact expected result, you usually have a reference answer: a correct answer you compare against, allowing for different wording.

The automated checker can be wrong. Comparing free-text answers by hand doesn't scale, so teams use another LLM to grade answers against rules. This is LLM-as-a-judge, and a model used this way is a judge. Think of a judge as a test oracle, the thing that decides whether a test passed, and the analogy works up to a point. You normally trust a test oracle. A judge is a model with its own error rate, so you have to measure how often it's right before its results mean anything. Part 2 shows how.

None of this makes AI untestable. You test it with the same discipline, plus some statistics and a way of checking the checker. A structured test of an AI system's quality is called an eval (short for evaluation), and that's what this series builds.

What RAG is

An LLM learns from a huge amount of text during training, then stops learning. It doesn't know your company's internal documents, or anything that happened after training. Ask it anyway and it may answer confidently from general knowledge, or simply make something up.

Retrieval-augmented generation (RAG) fixes that. Before the model answers, the system looks up relevant text in a set of documents and hands it to the model along with the question. The model is told to answer from what it was given. Lewis and colleagues introduced the approach in a 2020 paper [2], and it's now one of the most common ways companies put LLMs to work on their own data.

A RAG system has two halves:

• Retrieval finds the pieces of text most relevant to a question.
• Generation is the LLM writing an answer from the question plus those pieces.

The full set of documents the system can search is the corpus. One file in it is a document, and a piece of a document that retrieval returns is a passage. The passages handed to the model are its context.

One request flows like this:

1. A user asks a question.
2. The system converts the question into a form it can compare with the stored documents (more on that in the next section).
3. It searches the corpus and keeps the few passages that match best.
4. It builds a prompt: instructions, the retrieved passages, and the question.
5. The LLM reads the prompt and writes an answer.

Each step can fail in its own way. If you remember one idea from this article, make it this one.

Retrieval can miss: the passage with the answer is in the corpus, but it isn't among the passages returned. Or retrieval finds the right document but the wrong part of it.

Generation can fail even when retrieval did its job:

• Hallucination: the model states something that its context, or reality, doesn't support. The term is popular but loosely defined. For RAG, grounding is the more precise idea.
• Not grounded: a grounded answer is one where every claim is supported by the retrieved passages. An answer that adds facts from the model's general knowledge, or from a passage about something else, isn't grounded, even if the extra facts happen to be true.
• Misreading: the right passage is there, but the model takes the wrong number or credits an action to the wrong person.
• Refusal problems: a refusal is the system declining to answer, and sometimes that's correct. An unanswerable question is one the documents can't answer: the detail isn't there, the topic isn't covered, or the question rests on a false assumption. For those, the right behaviour is abstention: declining, or pointing out the false assumption. The opposite failure is refusing when the answer was in the passages all along.

Why evaluate the two halves separately? Because a wrong answer alone doesn't tell you what to fix. If retrieval never found the right passage, rewriting the model's instructions won't help. You know this principle already: a failing end-to-end test needs narrower tests behind it before you can locate the defect.

The building blocks you'll vary and test

A RAG system has a lot of settings, and each one changes its behaviour. These are the variables you'll record, pin, and sometimes change on purpose to test your evals. They fall into four groups.

Splitting documents into chunks

Documents are usually too long to search or to hand to a model whole, so the system cuts them into smaller pieces called chunks. That process is chunking. The chunk size is how long each piece is, often counted in characters or tokens. A token is the unit a model reads, roughly a word or part of a word.

Cut a sentence in half at a chunk boundary and the two halves may never be retrieved together. Chunk overlap repeats a little text from the end of each chunk at the start of the next, which lowers that risk.

Chunk size is a trade-off. Small chunks are precise but can lose the surrounding context. Large chunks carry more context, and more noise.

Finding the right passages

To search by meaning rather than exact words, the system turns each chunk into an embedding: a long list of numbers that represents what the text is about. Texts with similar meaning get similar numbers. The model that does this is the embedding model, or embedder. It's a separate model from the LLM that writes answers, usually much smaller, and swapping it changes what retrieval finds.

Embeddings live in a vector database, a store built to find the embeddings closest to a given one. At question time the question is embedded the same way, and a similarity search returns the closest chunks. Each result comes with a similarity score saying how close it is. A common measure is cosine similarity, where higher means more similar. Some systems report a distance instead, where lower means closer, so check which one yours uses before you set a threshold.

Two settings decide what's kept:

• Top-N (also written top-k) is how many of the best-scoring chunks go to the model, for example the top 4.
• The similarity threshold is a minimum score. Chunks below it are dropped, even if they're in the top N.

Set the threshold too strict and the model gets nothing. Too loose, and irrelevant text gets in.

Producing the answer

The text sent to the LLM is the prompt. Part of it is the system prompt: standing instructions from whoever configured the system, such as "Answer only from the provided documents. If they don't contain the answer, say so." The system prompt has a big effect on grounding and refusals.

Three model-level settings matter:

• Context window: the most text, in tokens, the model can read at once. The system prompt, passages and question all have to fit.
• Temperature: how much randomness goes into picking each next token. At 0 the model always takes the most likely continuation, which gives the most repeatable output. Higher values give more varied output. As mentioned earlier, 0 reduces variation but doesn't guarantee identical results.
• The model itself: a local model runs on your own hardware; a hosted model runs on a provider's servers and you call it over an API. Some models are reasoning (thinking) models: they write out intermediate reasoning before the answer. That's usually slower, and it changes what you need to record.

How the system behaves

Many RAG applications bundle a document set and its settings into a separate unit. AnythingLLM calls it a workspace. One workspace per configuration lets you compare configurations without them interfering with each other.

Most also let you pick a mode. In a document-only mode (AnythingLLM calls it query mode), the system answers only from retrieved context and returns a fixed refusal when nothing relevant turns up. In a conversational mode (chat mode), the model answers even with no relevant context, falling back on its general knowledge. If you're evaluating a system that's supposed to answer from documents, document-only mode matches the job. As a bonus, the fixed refusal text is something you can check exactly.

Here's an example configuration with the kind of settings you'll pin. The names and values are illustrative, not recommendations:

# Example only: one configuration of a RAG system under test
llm:
  model: example-model-8b
  temperature: 0
  context_window: 8192
embedder:
  model: example-embedder
chunking:
  size: 1000      # characters
  overlap: 20
retrieval:
  top_n: 4
  similarity_threshold: 0.25
generation:
  mode: query
  system_prompt: >-
    Answer only from the provided context. If the context does not contain
    the answer, say that the documents do not contain it.
  refusal_response: NO_RELEVANT_CONTEXT

Every line in that file can change an answer. That's why the coming stages treat configuration as part of the test, not as background.

The kinds of tools you'll need

You don't need a particular product to follow this series. You need four kinds of tools:

Role	What it does	Examples
RAG application or framework	Stores documents, retrieves passages and generates answers: the system under test	AnythingLLM, or an application built with LangChain or LlamaIndex
Model server	Runs the LLM and the embedding model	Ollama (runs models locally), or a hosted API
Evaluation framework	Runs metrics and judges over a set of test cases and collects the results	DeepEval, Ragas
Judge model	Grades answers against rules you write	Any capable model from a different family than the one you're testing

Running models locally has real advantages for evaluation. Calls cost nothing, your documents stay on your machine, and you can pin the exact model version so it doesn't change under you. The catch: you need hardware that runs the model at a usable speed, and long runs keep that hardware busy and hot.

An evaluation framework saves you writing plumbing: it organises test cases, runs metrics and reports results. What it can't do is tell you which failures your system actually has, or whether its built-in judges are any good on your data. Those calls are yours, and most of this series is about making them well.

Why should the judge come from a different model family? Research has found that LLM judges can recognise their own outputs and tend to rate them more favourably [5], so a model grading its own family's answers can be biased. Part 2 comes back to this.

My view: the tools matter much less than the method. Everything in this series carries over to a different stack.

The road ahead: ten stages

The rest of the series follows an evaluation from first contact with a system to an automated quality gate. Each stage builds on the one before.

Stage	Purpose	Part
0. Spike	Find out how the system actually behaves	1
1. Foundation	Make every run reproducible and every test case independent	1
2. Dataset	Build questions with known answers and evidence	2
3. Baseline runs and error analysis	See how the system really fails before choosing metrics	2
4. Evaluators	Build one check per failure that actually happens	2
5. Judge validation	Measure how often each AI judge is right	2
6. Statistics	Tell real differences from noise at small sample sizes	3
7. Mutations	Break the system on purpose to prove the evals notice	3
8. Eval gate and CI	Fail a change automatically when quality drops	3
9. Documentation and reproducibility	Make every number traceable and every decision explained	3

Each stage section has the same layout: the goal, why it matters, the key terms, how it's done, best practices and common mistakes.

Stage 0: Spike

Goal

Find out how the system under test really behaves before you design any evaluation around it.

Why it matters

Every eval rests on assumptions:

• that a setting does what its name says;
• that two test cases don't affect each other;
• that you can see what the model was given.

If one of these is false, your results will be wrong in ways that are hard to spot later. Finding out now is cheap. Finding out after weeks of evaluation costs you the weeks.

Key terms

A spike is a short, time-boxed investigation that answers a technical question before you commit to a design. The term comes from Extreme Programming, so you may know it from agile teams.

The system under test is the RAG application in one specific configuration, treated as a black box behind its interface, the same way you'd treat any service you test.

Test isolation means each test case runs without being affected by any other. With RAG systems this is less obvious than it sounds. Many applications keep chat history and feed earlier messages back into later ones. Some go further and reuse passages from earlier turns when retrieval comes up short, so the model sees text the current question never retrieved. And state can persist outside the conversation altogether: some applications cache answers to similar questions, keep a memory of facts across sessions, or reuse retrieval results, so even a brand-new session can be shaped by earlier cases.

How it's done

List the questions your evaluation depends on, then test each one against the real system and keep the evidence: requests, responses, screenshots. Typical questions:

1. Can you drive the system entirely through an API, without anyone clicking through a UI?
2. Which settings can you set and read back, and do changes actually take effect?
3. Does each response tell you which passages the model saw, and their scores?
4. If you send the same question twice at temperature 0, do you get the same answer?
5. What happens when nothing relevant is retrieved: a fixed refusal, or a guess?
6. Does conversation history carry over between questions, and can you start a fresh session for each one?
7. Does anything else persist between sessions, such as an answer cache, a memory feature or reused retrieval results, and can you turn it off or at least detect it?
8. Are there settings you can't control, such as a random seed, that will limit repeatability?

Here's the kind of experiment that pays off. Imagine you load a test system with two short, fictional documents, so any correct answer has to come from retrieval rather than the model's training. You ask a question whose answer exists in only one of them, with the similarity threshold set so high that nothing gets retrieved. In document-only mode you might get the fixed refusal text straight back. In conversational mode, the same setup might produce a confident, invented answer. One small experiment, and you know which mode to evaluate and what a refusal looks like in your traces.

A second experiment checks isolation. Run one question on its own, then run it again straight after a deliberately different question, and compare the passages the model received each time. The wording of the answers can vary anyway, but the passages should match; if they don't, something carried over from the earlier case. An answer cache won't show up this way, because it returns the same answer whatever came before. Look for it instead in cache-hit flags, near-zero response times, or repeated runs that come back word-for-word identical when they'd normally vary.

✅ Best practices

• Verify behaviour before designing evals. Every later stage assumes the system behaves as documented. Checking costs hours; finding a wrong assumption after weeks of evaluation costs the weeks.
• Use fictional test documents for the spike. The model can't have seen them in training, so a correct answer proves retrieval worked.
• Keep evidence for each finding. Saved requests and responses let you re-check a conclusion when the system is upgraded.
• Score retrieval on what the model actually received. Use the system's own record of the passages it passed to the model, not a separate search, so retrieval and generation are judged on the same input.

⚠️ Common mistakes

• Trusting setting names or documentation without testing them.
• Running test cases in one long conversation, so earlier questions influence later answers.
• Assuming temperature 0 means identical outputs, and treating a single run as the final word.
• Testing only the happy path, and never checking what happens when retrieval comes back empty.

Stage 1: Foundation

Goal

Make every evaluation run reproducible and every test case independent, so that a difference in results means a difference in the system, not in how the run was set up.

Why it matters

You'll compare runs constantly: before and after a prompt change, one model against another, this week's build against last month's. A comparison only means something if everything you didn't intend to change stayed the same. Without a foundation, a score drop could come from many sources:

• a model that was quietly updated;
• a setting someone changed in the UI;
• a test case that inherited history from the one before it.

Key terms

Configuration as code means the system's settings live in version-controlled files and are applied automatically, never edited by hand. It's the same idea as infrastructure as code. Provisioning is the step that applies a configuration: creating the workspace, loading the documents and setting every value from the file. Provisioning should be idempotent: running it twice leaves the same state as running it once.

A test case (usually just case) is one question with its expected outcome. A run is one execution of a configuration over a set of cases.

For each case, a run records a trace: everything you need to understand what happened. That's the question, the passages retrieved with their scores, the answer, the model that answered, the latency, the session ID, a cache-hit flag if the system reports one, and any error. Traces are what you'll read in Part 2 to find out how the system fails.

Each run also writes a manifest: a record of exactly what ran. It holds:

• the code version;
• the configuration and its hash (a short fingerprint that changes if any setting changes);
• the versions of every component;
• hashes of the dataset and documents.

Pinning means referring to an exact, unchangeable version of each component instead of a label that can move. Container images and many model registries give each version a digest, a content hash that identifies one exact version. A tag like latest can point to something different tomorrow [3].

The corpus needs care too. What an LLM knows from training is its parametric knowledge. If the model already knows the answers, a correct answer doesn't prove retrieval worked; it might have come from memory. When test material ends up in a model's training data, that's contamination (or leakage), and it quietly inflates scores [4]. Part 2 shows how to measure it directly with a closed-book baseline.

How it's done

1. Write the configuration as a file, like the example earlier, and provision the system from it. Each variant for an experiment is a new file that changes one setting, not a hand edit.

2. Run every case in a fresh session, so no history carries over. If the system offers threads or sessions, create a new one per case. Turn off any answer cache or memory feature the spike found, or record it in the trace if you can't.

3. Record a trace per case and a manifest per run.

An example trace record:

{
  "run_id": "2026-01-15T10:00:00Z-baseline",
  "case_id": "q-017",
  "session_id": "thread-8c41",
  "question": "Example question about a document in the corpus?",
  "retrieved": [
    {"doc_id": "report-12.pdf", "score": 0.81, "text": "Example passage text..."},
    {"doc_id": "report-12.pdf", "score": 0.77, "text": "Another passage..."}
  ],
  "answer": "Example answer text.",
  "model": "example-model-8b",
  "latency_s": 1.4,
  "cache_hit": false,
  "error": null
}

And an example manifest:

{
  "run_id": "2026-01-15T10:00:00Z-baseline",
  "git_commit": "a1b2c3d",
  "config_hash": "9f8e7d...",
  "app_image": "example/rag-app@sha256:5461...",
  "llm": {"name": "example-model-8b", "digest": "0aaa..."},
  "embedder": {"name": "example-embedder", "digest": "0a10..."},
  "dataset_sha256": "6dec...",
  "corpus_sha256": "bcb8..."
}

4. Pin every version. That covers:

• the application image, by digest;
• the models, by digest;
• your own dependencies, through a lockfile (a file that records the exact version of every library);
• for a hosted model, a dated model version rather than an alias that the provider can repoint.

Check the pins at the start of each run, and stop if anything doesn't match.

5. Choose the corpus deliberately. Look for documents the model is unlikely to have memorised, such as material published after the model's training data was collected. They should be deep enough to support questions that need several passages or several documents. And they should be licensed so you can use them for testing (and share them, if you plan to).

✅ Best practices

• Keep configuration as code. A setting that isn't in version control can't be reproduced or reviewed. Same reasoning as infrastructure as code.
• Isolate every test case. Shared history, caches and memory change what the model sees, so results stop being independent and traces stop being accurate (see Stage 0).
• Pin by digest, not by tag. A tag can be moved to a new version; a digest always means the same bytes [3].
• Write a manifest for every run. Any number you report later should trace back to the exact system that produced it.
• Choose a corpus that limits memorisation, and check the licence. Otherwise correct answers may reflect training data rather than retrieval [4].

⚠️ Common mistakes

• Changing settings by hand in a UI between runs, then not being able to say what changed.
• Using a latest tag for a model or application, so the system changes under you mid-project.
• Saving only the final answer, so you can't tell later whether a failure came from retrieval or generation.
• Testing on famous public documents (well-known articles, popular books) that the model has probably seen in training.

Recap and what's next

A RAG system retrieves passages from a corpus and writes an answer from them, and each half fails in its own way. Retrieval can miss the passage; generation can stray from it, misread it, or get refusals wrong.

Evaluating it takes the testing habits you already have, with three adjustments:

• outputs vary between runs;
• correctness is judged against a reference, not matched exactly;
• an automated judge has to be measured before you trust it.

Before writing a single metric, you spiked the system to confirm how it really behaves. Then you built a foundation where every setting is code, every case runs in isolation, every version is pinned, and every run leaves a trace and a manifest.

In Part 2: Building the Evals, you'll build a test dataset with known answers and evidence, read real failures to decide what to measure, write deterministic checks and AI judges for those failures, and measure how often each judge is right.

If you already test a RAG system, how do you handle these first two stages? Tell me in the comments.

References

1. He, H. and Thinking Machines Lab (2025). Defeating Nondeterminism in LLM Inference.
2. Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.
3. Docker documentation. Pull an image by digest (immutable identifier).
4. Sainz, O. et al. (2023). NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark.
5. Panickssery, A. et al. (2024). LLM Evaluators Recognize and Favor Their Own Generations.