Retrieval quality and benchmarks
How we measure retrieval quality, what the numbers mean, what they do not cover, and what they say about choosing a quality tier.
A knowledge base is only useful if it finds the right passage. We measure that continuously against public, independently built benchmarks rather than relying on our own impressions, and this page reports what those measurements say.
Every dataset below is public and openly licensed, so the numbers are quotable.
What we measure, and how
EnterpriseRAG-Bench
A realistic enterprise corpus: 5 722 documents in real folders, including misfiled documents and conflicting versions
CUAD
102 commercial contracts, asked both by naming the contract and by clause type
GermanQuAD
German language question answering across 474 documents
TAT-QA
Tables inside financial reports
Three things about the method matter for reading the results:
Every run goes through the real product. Documents are uploaded through the same interface you use, indexed by the same pipeline, and queried through the same chat and assistant paths. Nothing is simulated.
We compare like with like. A tier comparison is only valid when both knowledge bases hold the same documents. We measured that a half loaded collection scores about 15 points higher, simply because there is less to be wrong about, so every comparison here confirms the document count on both sides.
Answers are graded by a language model against the benchmark's own reference answer.
Reading the headline number
The main measure is recall@5: was the one passage the benchmark marked as correct among the top five results?
It is a strict measure and it understates real performance, for three reasons:
An AI assistant is given far more than five passages and answers from all of them. A correct passage in position nine is in front of the model and still counts as a miss here.
These benchmarks mark exactly one passage as correct. Real documents state the same fact in several places, and finding a different correct passage also counts as a miss.
It says nothing about the answer itself. That is what the answer quality figures below measure, and they are much higher.
We also report how highly the correct passage ranked, not just whether it appeared, because a result that is technically present but buried is a worse experience than the headline number suggests. On an easy collection the two can tell opposite stories, so we look at both.
Treat differences under about three points as noise. We measured the pipeline's own run to run variation by scoring the identical collection three times with nothing meaningful changed, and it moved by up to 3.4 points in both directions.
Answer quality
The figures closest to what you actually experience.
EnterpriseRAG-Bench
0.76
1.00
CUAD, by contract name
0.71 to 0.73
0.98
CUAD, by clause type
0.90 to 0.95
0.99
GermanQuAD
0.93 to 0.97
0.88 to 0.94
TAT-QA
0.75 to 0.83
0.97 to 1.00
Answer relevance sits between 0.88 and 1.00 everywhere, meaning the assistant addresses the question it was asked. Groundedness, which asks whether the answer stays anchored to the retrieved documents rather than drifting, is the more demanding measure and ranges from 0.71 to 0.97 depending on how hard the documents are.
Results by tier
Recall@5 for a direct search of the knowledge base.
GermanQuAD
1.00
1.00
0.99
CUAD, by clause type
0.99
0.98
1.00
TAT-QA
0.93
0.93
0.94
EnterpriseRAG-Bench
0.90
0.91
not yet measured
CUAD, by contract name
0.80
0.82
0.88
And when an AI assistant searches for itself, which returns many more passages and is the harder test:
GermanQuAD
0.98
0.98
0.99
CUAD, by contract name
0.86
0.92
0.83
CUAD, by clause type
0.75
0.73
0.76
TAT-QA
0.70
0.63
0.72
EnterpriseRAG-Bench
0.70
0.69
not yet measured
What the results actually say
No tier wins everywhere
Each of the three wins somewhere and loses somewhere. There is no single best configuration, only one that suits a given kind of document. That is why the tier control exists, and why the guidance on Quality tiers and advanced settings is written around document types rather than around a ranking.
An easy collection tells you nothing about your tier choice
GermanQuAD is the clearest case. All three tiers land between 0.98 and 1.00 on both paths, inside the noise floor of each other. On a collection of clean encyclopaedia text with a well posed question, every tier finds the right document, and the tier control is not what decides your result.
The clause type results say the same thing. This is worth knowing before you pay for a higher tier: if your documents are straightforward text, the cheapest tier is very likely enough.
Where the ranking measure disagrees with recall
On GermanQuAD the three tiers are level on whether the right document is found, and not level on where it lands in the list. Precision ranks it slightly lower than the other two on a broad assistant search, and slightly higher on a narrow direct search.
That is the knowledge graph at work, and it matches what we measured deliberately (next section). It is also why this page reports both measures: on an easy collection, recall alone would have said the tiers are identical, which is not true.
Table handling, and a conclusion we had to withdraw
TAT-QA is tables inside financial reports, and the three tiers read tables three different ways.
Standard
Fast
0.70
Enhanced
Balanced
0.63
Maximum
Precision
0.72
It is tempting to read that as "Enhanced is the weak setting", and an earlier draft of this page said exactly that. We tested it directly and it was wrong. Running the middle tier with the Standard table setting, everything else held identical, scored 0.64, which is statistically the same as its 0.63 with Enhanced. Changing the table setting did not recover the gap, so the table setting is not what produced it.
What survives is narrower and still useful:
Maximum leads on tables, consistent with it reading them from an image of the page.
If tables are not your bottleneck, changing this setting alone will not move your results. We do not yet know what accounts for the middle tier's score on this benchmark, and we would rather say so than repeat a tidy explanation we have disproved.
The knowledge graph changes ranking, not what is found
We tested this directly, running the same documents and the same questions with the knowledge graph contribution to ranking switched off.
It almost never changed whether the correct passage was found. It changed where in the list it appeared, and the direction depended on how many passages were being considered:
For a narrow, precise search it improved the ranking by about 16 points.
For a broad search considering many passages it worsened the ranking, by up to 13 points.
We have now seen the same pattern on three separate benchmarks, including one measured a different way, so we are confident it is real rather than a quirk of one corpus.
So the knowledge graph is worth switching into retrieval for precise, entity heavy collections such as contracts, and worth leaving off for general ones.
Reranking was the single highest value improvement
Adding a reranking model, a second model that re-reads the candidate passages against the question and re-orders them, improved direct search results on one benchmark by 26 percentage points. It costs nothing at indexing time and one call per question, and it now runs on every tier.
On clean text collections, the cheapest tier holds its own
EnterpriseRAG-Bench is the largest corpus here: 5 722 documents with genuine folder structure, misfiled documents and conflicting versions. Every difference between Fast and Balanced falls inside the noise floor.
Worth stating plainly rather than hiding: on collections of ordinary machine readable text, paying for per passage AI analysis is not automatically worth it. Read the next section before applying that finding to your own documents, because it is a statement about this corpus and not about yours.
The cost side of that trade is measured on Quality tiers and advanced settings: on plain text the middle tier indexes at roughly ten times the cost of the cheapest, while on documents with images or scans the tiers cost within a factor of two of each other, because reading the page is most of the bill on every tier.
What these benchmarks do not tell you
Public benchmarks are the honest way to measure retrieval, because we did not build them and cannot tune against them. They come with one significant blind spot, and it matters most for the Precision tier.
These corpora are easier to read than your documents
Almost everything above is born digital, machine readable text: synthetic office documents, contract text, encyclopaedia articles. Of the four, only the financial reports set genuinely stresses tables. None of them stress the things that actually make customer documents hard:
Scanned pages and photographed documents
Dense multi column layouts
Engineering drawings, schematics and figures that carry the answer
Tables that span pages, or that sit inside an image
Handwriting, stamps and annotations
Several languages inside one document
We do measure those, on datasets built for exactly that purpose. Their licences permit research and internal use but not publication, so their scores cannot appear on this page. That is a licensing limit rather than an absence of testing, and it is the reason this page looks tidier than the underlying reality.
Which means these numbers probably understate Precision
This is the important consequence, so it is worth being direct about it.
Everything the Precision tier adds over Balanced is aimed at documents that are hard to read: reading complex tables from an image of the page, meaning based passage boundaries, and, in the configuration we benchmark, stronger OCR and vision models. On a corpus of clean digital text none of that has anything to do. The tier pays its full cost and has no opportunity to show its advantage, so it lands roughly level with the cheaper tiers.
That is exactly the pattern above. Precision ties or leads on every clean text benchmark by a margin too small to matter, and leads clearly only on tables, the one published corpus that puts any of its strengths to work.
So please do not read "the cheapest tier is enough" as a general statement about your collection. It is a statement about corpora of clean text. If your documents are scanned contracts, photographed forms, technical manuals full of figures, or reports whose numbers live in complex tables, the gap between the tiers on your data can be considerably larger than the tables above suggest, and in Precision's favour.
A benchmark can also measure the wrong thing
One more caution, because it bit us while preparing this page. Some benchmarks judge a result by matching a retrieved passage exactly against their own reference passage. That rewards a tier which happens to cut passages the same way the benchmark did, and penalises one that cuts them differently, even when it retrieved exactly the right content.
On GermanQuAD that scoring method put Precision 13 points below the other tiers. Re-measured on a like for like basis, Precision was in fact marginally ahead. The 13 point gap was measuring passage boundaries, not retrieval quality.
We mention it because it is the kind of error that makes benchmark tables misleading in a direction nobody notices, and because it is a further reason to test on your own documents rather than trust a table.
Precision is also the least measured tier
It has been measured on four of the five benchmarks above, against all five for the other two tiers, because its indexing runs at roughly two documents per minute and a full measurement is therefore slow and expensive. Where a cell says "not yet measured", that is exactly what it means, and it should not be read as a poor result.
How to settle it for your own documents
Benchmarks are for choosing a starting point. They cannot settle the question on your data, and for hard to read documents they are not even a good guide. If that describes your collection:
Put a representative sample into two knowledge bases, one on Balanced and one on Precision. Include your most awkward documents rather than your tidiest.
Ask the same ten or twenty real questions against both.
Compare the answers and the passages they cite, not just whether an answer appeared.
That comparison, on your own documents, is worth more than any number on this page. Your account team can help you set it up.