> For the complete documentation index, see [llms.txt](https://docs.blockbrain.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.blockbrain.ai/for-users/all-about-knowledge-management/retrieval-quality-and-benchmarks.md).

# Retrieval quality and benchmarks

How we measure retrieval quality, what the numbers mean, what they do not cover, and what they say about choosing a quality tier.

A knowledge base is only useful if it finds the right passage. We measure that continuously against public, independently built benchmarks rather than relying on our own impressions, and this page reports what those measurements say.

Every dataset below is public and openly licensed, so the numbers are quotable.

{% hint style="info" %}
**Read** [**What these benchmarks do not tell you**](#what-these-benchmarks-do-not-tell-you) **before drawing conclusions for your own collection.** These corpora are easier to read than most real customer documents, which systematically understates the Precision tier.
{% endhint %}

## What we measure, and how

| Benchmark               | What it tests                                                                                                         |
| ----------------------- | --------------------------------------------------------------------------------------------------------------------- |
| **EnterpriseRAG-Bench** | A realistic enterprise corpus: 5 722 documents in real folders, including misfiled documents and conflicting versions |
| **CUAD**                | 102 commercial contracts, asked both by naming the contract and by clause type                                        |
| **GermanQuAD**          | German language question answering across 474 documents                                                               |
| **TAT-QA**              | Tables inside financial reports                                                                                       |

Three things about the method matter for reading the results:

* **Every run goes through the real product.** Documents are uploaded through the same interface you use, indexed by the same pipeline, and queried through the same chat and assistant paths. Nothing is simulated.
* **We compare like with like.** A tier comparison is only valid when both knowledge bases hold the same documents. We measured that a half loaded collection scores about 15 points **higher**, simply because there is less to be wrong about, so every comparison here confirms the document count on both sides.
* **Answers are graded by a language model** against the benchmark's own reference answer.

### Reading the headline number

The main measure is **recall\@5**: was the one passage the benchmark marked as correct among the top five results?

**It is a strict measure and it understates real performance**, for three reasons:

1. An AI assistant is given far more than five passages and answers from all of them. A correct passage in position nine is in front of the model and still counts as a miss here.
2. These benchmarks mark exactly one passage as correct. Real documents state the same fact in several places, and finding a different correct passage also counts as a miss.
3. It says nothing about the answer itself. That is what the answer quality figures below measure, and they are much higher.

We also report **how highly** the correct passage ranked, not just whether it appeared, because a result that is technically present but buried is a worse experience than the headline number suggests. On an easy collection the two can tell opposite stories, so we look at both.

**Treat differences under about three points as noise.** We measured the pipeline's own run to run variation by scoring the identical collection three times with nothing meaningful changed, and it moved by up to 3.4 points in both directions.

## Answer quality

The figures closest to what you actually experience.

| Benchmark              | Answer stays grounded in your documents | Answer addresses the question |
| ---------------------- | --------------------------------------- | ----------------------------- |
| EnterpriseRAG-Bench    | 0.76                                    | 1.00                          |
| CUAD, by contract name | 0.71 to 0.73                            | 0.98                          |
| CUAD, by clause type   | 0.90 to 0.95                            | 0.99                          |
| GermanQuAD             | 0.93 to 0.97                            | 0.88 to 0.94                  |
| TAT-QA                 | 0.75 to 0.83                            | 0.97 to 1.00                  |

Answer relevance sits between 0.88 and 1.00 everywhere, meaning the assistant addresses the question it was asked. Groundedness, which asks whether the answer stays anchored to the retrieved documents rather than drifting, is the more demanding measure and ranges from 0.71 to 0.97 depending on how hard the documents are.

## Results by tier

Recall\@5 for a direct search of the knowledge base.

| Benchmark                  | Fast     | Balanced | Precision        |
| -------------------------- | -------- | -------- | ---------------- |
| **GermanQuAD**             | **1.00** | **1.00** | 0.99             |
| **CUAD, by clause type**   | 0.99     | 0.98     | **1.00**         |
| **TAT-QA**                 | 0.93     | 0.93     | **0.94**         |
| **EnterpriseRAG-Bench**    | 0.90     | **0.91** | not yet measured |
| **CUAD, by contract name** | 0.80     | 0.82     | **0.88**         |

And when an AI assistant searches for itself, which returns many more passages and is the harder test:

| Benchmark                  | Fast     | Balanced | Precision        |
| -------------------------- | -------- | -------- | ---------------- |
| **GermanQuAD**             | 0.98     | 0.98     | **0.99**         |
| **CUAD, by contract name** | 0.86     | **0.92** | 0.83             |
| **CUAD, by clause type**   | 0.75     | 0.73     | **0.76**         |
| **TAT-QA**                 | 0.70     | 0.63     | **0.72**         |
| **EnterpriseRAG-Bench**    | **0.70** | 0.69     | not yet measured |

## What the results actually say

### No tier wins everywhere

Each of the three wins somewhere and loses somewhere. There is no single best configuration, only one that suits a given kind of document. That is why the tier control exists, and why the guidance on Quality tiers and advanced settings is written around document types rather than around a ranking.

### An easy collection tells you nothing about your tier choice

GermanQuAD is the clearest case. All three tiers land between 0.98 and 1.00 on both paths, inside the noise floor of each other. On a collection of clean encyclopaedia text with a well posed question, **every tier finds the right document**, and the tier control is not what decides your result.

The clause type results say the same thing. This is worth knowing before you pay for a higher tier: if your documents are straightforward text, the cheapest tier is very likely enough.

### Where the ranking measure disagrees with recall

On GermanQuAD the three tiers are level on **whether** the right document is found, and not level on **where it lands in the list**. Precision ranks it slightly lower than the other two on a broad assistant search, and slightly higher on a narrow direct search.

That is the knowledge graph at work, and it matches what we measured deliberately (next section). It is also why this page reports both measures: on an easy collection, recall alone would have said the tiers are identical, which is not true.

### Table handling, and a conclusion we had to withdraw

TAT-QA is tables inside financial reports, and the three tiers read tables three different ways.

| Table setting | Tier      | Recall\@5, assistant search |
| ------------- | --------- | --------------------------- |
| Standard      | Fast      | 0.70                        |
| Enhanced      | Balanced  | 0.63                        |
| Maximum       | Precision | **0.72**                    |

It is tempting to read that as "Enhanced is the weak setting", and an earlier draft of this page said exactly that. **We tested it directly and it was wrong.** Running the middle tier with the **Standard** table setting, everything else held identical, scored **0.64**, which is statistically the same as its 0.63 with Enhanced. Changing the table setting did not recover the gap, so the table setting is not what produced it.

What survives is narrower and still useful:

* **Maximum leads on tables**, consistent with it reading them from an image of the page.
* **If tables are not your bottleneck, changing this setting alone will not move your results.** We do not yet know what accounts for the middle tier's score on this benchmark, and we would rather say so than repeat a tidy explanation we have disproved.

### The knowledge graph changes ranking, not what is found

We tested this directly, running the same documents and the same questions with the knowledge graph contribution to ranking switched off.

It almost never changed **whether** the correct passage was found. It changed **where in the list** it appeared, and the direction depended on how many passages were being considered:

* For a narrow, precise search it improved the ranking by about 16 points.
* For a broad search considering many passages it worsened the ranking, by up to 13 points.

We have now seen the same pattern on three separate benchmarks, including one measured a different way, so we are confident it is real rather than a quirk of one corpus.

So the knowledge graph is worth switching into retrieval for precise, entity heavy collections such as contracts, and worth leaving off for general ones.

### Reranking was the single highest value improvement

Adding a reranking model, a second model that re-reads the candidate passages against the question and re-orders them, improved direct search results on one benchmark by 26 percentage points. It costs nothing at indexing time and one call per question, and it now runs on every tier.

### On clean text collections, the cheapest tier holds its own

EnterpriseRAG-Bench is the largest corpus here: 5 722 documents with genuine folder structure, misfiled documents and conflicting versions. Every difference between Fast and Balanced falls inside the noise floor.

Worth stating plainly rather than hiding: **on collections of ordinary machine readable text, paying for per passage AI analysis is not automatically worth it.** Read the next section before applying that finding to your own documents, because it is a statement about this corpus and not about yours.

The cost side of that trade is measured on Quality tiers and advanced settings: on plain text the middle tier indexes at roughly ten times the cost of the cheapest, while on documents with images or scans the tiers cost within a factor of two of each other, because reading the page is most of the bill on every tier.

## What these benchmarks do not tell you

Public benchmarks are the honest way to measure retrieval, because we did not build them and cannot tune against them. They come with one significant blind spot, and it matters most for the Precision tier.

### These corpora are easier to read than your documents

Almost everything above is **born digital, machine readable text**: synthetic office documents, contract text, encyclopaedia articles. Of the four, only the financial reports set genuinely stresses tables. None of them stress the things that actually make customer documents hard:

* Scanned pages and photographed documents
* Dense multi column layouts
* Engineering drawings, schematics and figures that carry the answer
* Tables that span pages, or that sit inside an image
* Handwriting, stamps and annotations
* Several languages inside one document

We do measure those, on datasets built for exactly that purpose. Their licences permit research and internal use but not publication, so their scores cannot appear on this page. **That is a licensing limit rather than an absence of testing**, and it is the reason this page looks tidier than the underlying reality.

### Which means these numbers probably understate Precision

This is the important consequence, so it is worth being direct about it.

Everything the Precision tier adds over Balanced is aimed at documents that are hard to read: reading complex tables from an image of the page, meaning based passage boundaries, and, in the configuration we benchmark, stronger OCR and vision models. On a corpus of clean digital text **none of that has anything to do**. The tier pays its full cost and has no opportunity to show its advantage, so it lands roughly level with the cheaper tiers.

That is exactly the pattern above. Precision ties or leads on every clean text benchmark by a margin too small to matter, and leads clearly only on tables, the one published corpus that puts any of its strengths to work.

So please do not read "the cheapest tier is enough" as a general statement about your collection. It is a statement about corpora of clean text. If your documents are scanned contracts, photographed forms, technical manuals full of figures, or reports whose numbers live in complex tables, the gap between the tiers on your data can be considerably larger than the tables above suggest, and in Precision's favour.

### A benchmark can also measure the wrong thing

One more caution, because it bit us while preparing this page. Some benchmarks judge a result by matching a retrieved passage **exactly** against their own reference passage. That rewards a tier which happens to cut passages the same way the benchmark did, and penalises one that cuts them differently, even when it retrieved exactly the right content.

On GermanQuAD that scoring method put Precision 13 points below the other tiers. Re-measured on a like for like basis, Precision was in fact marginally **ahead**. The 13 point gap was measuring passage boundaries, not retrieval quality.

We mention it because it is the kind of error that makes benchmark tables misleading in a direction nobody notices, and because it is a further reason to test on your own documents rather than trust a table.

### Precision is also the least measured tier

It has been measured on four of the five benchmarks above, against all five for the other two tiers, because its indexing runs at roughly two documents per minute and a full measurement is therefore slow and expensive. Where a cell says "not yet measured", that is exactly what it means, and it should not be read as a poor result.

### How to settle it for your own documents

Benchmarks are for choosing a starting point. They cannot settle the question on your data, and for hard to read documents they are not even a good guide. If that describes your collection:

1. Put a representative sample into two knowledge bases, one on Balanced and one on Precision. Include your most awkward documents rather than your tidiest.
2. Ask the same ten or twenty real questions against both.
3. Compare the answers **and the passages they cite**, not just whether an answer appeared.

That comparison, on your own documents, is worth more than any number on this page. Your account team can help you set it up.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.blockbrain.ai/for-users/all-about-knowledge-management/retrieval-quality-and-benchmarks.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.