Why RAG evaluation must check more than document retrieval

RAG evaluation must check how retrieved documents shape answers

An AI system can have the correct financial statement available and still answer from a statement for a different year. This creates an evaluation problem for teams building retrieval-augmented generation, or RAG, systems, which retrieve documents for a model to use when answering. Confirming that the intended file reached the model establishes access. Teams also need to check source fidelity: whether the answer reflects the intended record.

Seekr’s study of how large language models use retrieved documents examines this gap. The company found that near-identical records can compete to shape an answer, producing a clear, confident response based on the wrong version. Seekr has a commercial interest in businesses investing in AI and its evaluation, so readers should weigh its findings and recommendations with that stake in mind.

Seekr tested two models under three document conditions

To examine how similar records affect answers, Seekr used US Treasury financial statements in its study, published under the title “Study finds AI can use wrong document despite retrieval.” Seekr chose these statements because they are public, allow answers to be checked objectively and resemble closely related records in business and government collections. Ben Faircloth, the paper’s author and Seekr’s leader of AI strategy and research, was listed in the company’s announcement as available to discuss the methodology.

Using those statements, Seekr assembled 80 questions and tested Llama-3.3-70B and Qwen2.5-72B, referred to here as Llama and Qwen. Each model faced three document conditions: the correct statement alone; that statement alongside similar statements from other years; and a broader retrieval setting that searched across all the statements. This let Seekr compare a straightforward document-use task with conditions where competing records could affect selection or answering.

To distinguish those effects, Seekr measured answer accuracy and used attribution, a method for tracking which document shaped an answer. Accuracy recorded whether the answer was correct, while attribution helped identify cases where a competing record influenced it. Together, these measurements let Seekr investigate failures during retrieval and failures after the intended document reached the model.

Similar statements reduced accuracy while the correct record remained available

The first condition established how the models performed with only the intended statement. Seekr reported accuracy of 95.0% for Llama and 92.5% for Qwen, showing that both answered most questions correctly. These results provided the baseline for measuring what changed when similar records appeared beside the correct one.

Adding statements from other years reduced accuracy even though the intended record remained available. Seekr reported that Llama’s accuracy fell to 85.0%, while Qwen’s fell to 87.5%. The company described the decline as 5 to 10 percentage points in a condition where retrieval had already succeeded. Accuracy fell for both model families, though by different amounts.

Because the correct statement was present in both conditions, its absence cannot explain this decline. A document-access check would have passed in each case, yet the answers became less accurate when competing statements were added. The controlled comparison therefore establishes a risk of incorrect answers after successful retrieval, while leaving the cause of each error to further investigation.

Attribution helped Seekr investigate those causes by identifying cases where a model appeared to rely on a competing document despite having the correct one available. A statement from another year can contain plausible financial information while describing the wrong reporting period. This explains how an answer can remain fluent while drawing on an unsuitable record, without assuming that every error in the second condition had the same cause.

Broader retrieval introduced a larger record-selection problem

The controlled comparison kept the intended statement available, but the broader retrieval condition required the system to find it first. When the system searched across all statements, Seekr reported accuracy on answered questions of 37.1% for Llama and 30.4% for Qwen. Seekr attributed most of the full-pipeline performance gap, the decline across document retrieval and answer generation, to retrieval selecting the wrong record. Record selection was therefore the largest reported problem in that setting.

Because the broader condition combined retrieval with answering, its figures cannot directly measure confusion after successful retrieval. An error could begin when the system selected a record or arise later when the model used the available material. The second condition isolated the latter risk more clearly by keeping the correct statement present, so the comparisons help diagnose different problems.

The distinction matters when deciding what to improve. Seekr argues that a right-or-wrong answer score cannot establish whether an error came from retrieval, reasoning or reliance on the wrong document among similar records. Each explanation points to a different part of the system to investigate. Improving retrieval addresses the largest reported problem in the broader test, while the controlled results give teams a separate reason to examine how models use documents they already have.

Evaluation should track access, source use and correctness

To investigate these different ways a system can fail, engineering teams can ask three questions: Was the intended record retrieved? Did the answer reflect that record? Was the answer correct? Keeping the questions separate preserves evidence about where a failure occurred.

Separating these checks extends the accuracy-focused evaluation that Seekr says many businesses have used for AI systems. Accuracy establishes whether a response is right, while a retrieval check establishes whether the intended material was available. Attribution can then help investigate which material shaped the response. Recording these observations separately helps evaluators distinguish a selection error from a source-use error or a reasoning error.

These distinctions become clearer in collections with several versions of a record. Seekr places the issue within the evaluation of systems using proprietary or regulated data, particularly in finance, legal work, compliance and the public sector. Such collections can resemble the Treasury material because closely related documents contain small differences that determine which answer fits the question. The relevant test is whether the response uses the intended revision or reporting period.

For a legal team, that could mean distinguishing contract revisions and amended contracts; for a financial team, it could mean distinguishing quarterly filings, updated filings and annual statements. Government and compliance teams face a similar issue with regulatory papers. When these records coexist, an evaluation needs to identify the intended version and examine whether the answer reflects it, because plausible information from a neighbouring document can still answer the wrong question.

Key takeaways for decision-makers

• Test competing document versions: Seekr tested two models on 80 questions under three document conditions. AI evaluation owners can use this structure to compare performance with the intended record alone, alongside similar records and within a broader retrieval pipeline.
• Check which source shapes the answer: Adding similar statements reduced accuracy by 5 to 10 percentage points despite the correct record remaining available. Engineering teams need source-use checks to identify when competing versions influence answers.
• Prioritize record selection in retrieval: Accuracy on answered questions fell to 37.1% for Llama and 30.4% for Qwen in the broader retrieval test. Seekr attributed most of that performance gap to selecting the wrong record, making retrieval selection the first area to investigate in that setting.
• Separate evaluation signals: Platform teams need distinct checks for intended-record retrieval, source use and answer correctness. Recording these signals separately helps identify whether improvements belong in retrieval, document handling or reasoning.