Every RAG write-up leads with the pipeline: the embedding model, the graph, the reranker. Almost none of them lead with the thing that decides whether any of it worked - the benchmark. For FinLaw-UK, my MSc dissertation on UK financial regulation QA, nobody hands you an eval set. I had to write one, and building it taught me more about where retrieval systems fail than building the retrieval did.
What one benchmark item has to carry
A useful legal QA item is more than a question and an answer. Each item in the FinLaw-UK set carries four things: the question, a gold answer, the citations a correct answer must include, and the keywords the grader checks for. Here is a real one from the curated tier:
- Question: "Which FCA principle introduces the Consumer Duty?"
- Gold answer: the Consumer Duty is Principle 12, requiring firms to act to deliver good outcomes for retail customers.
- Required citation: PRIN 12.
- Graded keywords: Consumer Duty, good outcomes, Principle 12.
The required-citation field is the one that earns its keep. The pipeline validates citations against a Neo4j knowledge graph of the regulatory corpus, so an answer citing a rule that does not exist in the graph is flagged as a potential hallucination. The benchmark mirrors that: an answer without PRIN 12 in it is wrong even if the prose is right, because in regulatory work an uncited claim is not an answer - it is a liability.
Composition matters more than size
The full set is 110 items, and the split matters more than the count: 80 short-form questions, 20 document-level tasks, and 10 case-like scenarios, spanning domains from FSMA authorisation and conduct-of-business rules to payment services and complaints handling.
Each tier probes a different failure mode. Short-form questions check retrieval precision: can the system find the one rule that answers a pointed question? Document tasks check synthesis: can it hold a whole chapter in view without dropping the cross-references? Scenarios check application: given a fact pattern, does it reach for the right regime at all? A benchmark that is all short-form questions will happily certify a system that falls over the moment a question spans two documents.
The aggregate score hid a broken scorer
The headline numbers looked healthy: 0.82 source accuracy, 0.81 citation quality, 0.76 RAGAS faithfulness, 0.74 answer relevance. I submitted on those. Then I went back to reproduce them and found that two of the four were not measuring anything.
The source-accuracy and citation-quality scorers were regex shape-checks. They tested whether a model output contained something shaped like a citation, not whether that citation was the right one, or real. Anything citation-shaped scored a flat 0.85. Once I looked at the distribution instead of the mean, 103 of the 110 rows carried that identical constant. The true rate, when every citation is resolved against the graph, was 3 in 110.
A mean can look reasonable while the distribution underneath it is a single repeated number. Plot the spread before you quote the average.
Both figures are withdrawn. The repo now ships a measurement-integrity report saying which numbers reproduce and which do not, because a benchmark you release without that is worth less than no benchmark at all. The correction is also what motivated the parts of the system I am proudest of: the citation normaliser, the cross-encoder re-ranker, and the refusal gate that returns nothing rather than a citation it cannot verify.
Refusing costs something, and the benchmark is what let me price it. Headline answer relevancy drops 23 points, because RAGAS scores a refusal as zero by construction. Exclude the 30 refusals and the mean is 0.658, against 0.641 for the non-refusing baseline. The baseline only scored higher by confabulating answers to questions the corpus cannot support, which is precisely the behaviour the gate exists to stop.
Where it actually failed
The per-domain breakdown told a different story in one corner: consumer redress. DISP - the complaints-handling sourcebook - dragged legal completeness below 0.65. The reason is structural. DISP rules are fragmented into many small, heavily cross-referenced fragments, and the two-hop graph expansion that stitches context back together for the rest of the Handbook could not reassemble them. Rule-level chunking is the right default for most of the corpus; for DISP it shreds the very context the expansion is supposed to recover.
That is a finding I only have because the benchmark was built to expose it: domain labels on every item meant the failure had an address, not just a lower average.
What I would build next
Two things. Domain-aware chunk granularity for fragmented sourcebooks like DISP, so completeness recovers where the structure is hostile and deeper expansion alone will not do it. And a structured legal-reasoning benchmark beyond RAGAS, because faithfulness scoring cannot catch an answer that is perfectly grounded in a retrieved passage and still misapplies it across jurisdictions. The eval set is public in the repo alongside the pipeline and the integrity report; if you are building anything similar, steal the format - question, gold answer, required citations, graded keywords. The fourth column is the one you will thank yourself for, and it is the one that catches a scorer measuring shape instead of substance.