RAG + Quality Engineering

How to Evaluate a RAG System: A QA Engineer’s Practical Test Plan

Learn how to test retrieval quality, grounded answers, refusal behaviour and production reliability in a RAG system—before trusting a polished demo.

By Rakesh Kumar Singh2026-07-3011 min read

Why a convincing RAG demo can still fail

A retrieval-augmented generation system can answer five prepared questions beautifully and still be unsafe for real users. The demo may use familiar wording, clean documents and questions whose answers appear in one obvious paragraph. Production traffic introduces misspellings, vague language, conflicting versions, missing evidence and questions that should not be answered at all.

From teaching RAG and working in quality engineering, I find the most useful mental shift is to stop testing the chatbot as one black box. A RAG response is the result of at least two important stages: retrieving evidence and generating an answer from that evidence. Each stage needs its own test evidence.

The original RAG research described a model that combines parametric memory with an explicit non-parametric memory. That separation is practical for testers: first ask whether the system found the right material, then ask whether the model used it faithfully.

Step 1: define what a correct answer means

Begin with a small evaluation set created from documents you trust. Each test case should contain a realistic user question, the document or passage that should support the answer, the essential facts expected in the response and any conditions that should trigger a refusal or clarification.

Do not build the set only from headings or copy sentences directly from the source. Include the ways people actually ask: short questions, synonyms, abbreviations, indirect wording and questions containing an incorrect assumption. Add a few questions whose answers do not exist in the knowledge base.

For a leave-policy assistant, for example, one test might ask, “Can I move unused casual leave to next year?” The expected evidence should identify the exact policy clause. A second test could ask about a benefit that the policy never mentions; the expected behaviour is to say that the available sources do not establish an answer.

Step 2: test retrieval before reading the final answer

Record the top passages returned for every test question. Recall at k asks whether at least one required passage appeared within the first k results. Precision at k asks how much of that retrieved set was actually relevant. Mean reciprocal rank rewards systems that place the first useful result nearer the top.

These metrics reveal different defects. Low recall may point to poor chunk boundaries, missing documents, weak query wording or an unsuitable retrieval method. Acceptable recall with poor ranking may mean useful evidence is being buried beneath similar but irrelevant passages.

Do not assume a dense vector search is automatically the strongest option. The BEIR benchmark found that retrieval performance varies across domains and reported BM25 as a robust baseline, while stronger re-ranking approaches can add computational cost. Compare a simple baseline, vector retrieval and a hybrid or reranked approach on your own questions.

Step 3: test whether the answer is grounded

Once retrieval is reliable, evaluate the generated response. Check factual correctness against the supplied passage, completeness against the expected facts and groundedness: every important claim should be supported by the retrieved evidence.

Also check citation usefulness. A source label is not enough if it points only to a long document. The citation should help a person locate the supporting section, and the quoted or summarized claim must match that section.

Use human review for a representative sample, especially when the answer affects money, health, employment or access. An automated model-based judge can help scale triage, but it should use a clear rubric, return reasons and be calibrated against human decisions rather than treated as the truth.

Step 4: design negative and adversarial cases

A trustworthy test plan includes questions the system cannot answer. Remove the relevant document, ask about a future policy, combine two departments’ rules, use a misleading premise and request a conclusion that the evidence does not support. The desired result may be a careful limitation, a clarifying question or escalation to a person.

Test conflicting evidence deliberately. If an old and new policy are both indexed, does the system prefer the effective version and disclose the date? Test document-level instructions too: retrieved text may contain sentences that look like commands to the model. Knowledge content should be treated as evidence, not as authority to override system rules.

These tests protect against false confidence. A shorter answer that says what is unknown is often more useful than a fluent answer assembled from weak clues.

Step 5: test the production system, not only the model

Measure response time, failure rate, retrieval latency, token usage and cost for realistic workloads. Check document updates, deletion, access controls and stale indexes. If one user cannot open a document, retrieval must not expose its contents through an answer.

Monitor quality after release with privacy-safe traces: the question category, retrieved document identifiers, ranking scores, refusal behaviour, user feedback and the version of the index or prompt. NIST’s Generative AI Profile emphasises ongoing measurement, monitoring and periodic review because risks and performance can change after deployment.

Create a regression suite from real failures. Every confirmed retrieval miss, unsupported claim or confusing refusal should become a repeatable test before the next release.

A practical release gate for a small RAG project

A useful first gate is not “the chatbot looks good.” Require an agreed retrieval target on a representative test set, no unsupported high-impact claims in the reviewed sample, correct behaviour for unanswered questions, verified access control and acceptable latency and cost.

Keep the numbers tied to the use case. A learning assistant and a financial decision-support tool do not deserve the same risk tolerance. Document who approved the thresholds and what happens when monitoring falls below them.

The most important habit is simple: diagnose the failed stage. If retrieval missed the evidence, work on documents, chunks, queries or ranking. If the evidence was present but the answer was wrong, work on instructions, context use and response validation. That turns RAG improvement from prompt guessing into quality engineering.

Continue with a useful next step

Primary sources

These references support the technical and risk-management points above.

What should I test or explain next?

Your real question can become a future experiment and a useful public lesson.

Suggest a problem