Every RAG write-up leads with the pipeline: the embedding model, the graph, the reranker. Almost none of them lead with the thing that decides whether any of it worked - the benchmark. For FinLaw-UK, my MSc dissertation on UK financial regulation QA, nobody hands you an eval set. I had to write one, and building it taught me more about where retrieval systems fail than building the retrieval did.
What one benchmark item has to carry
A useful legal QA item is more than a question and an answer. Each item in the FinLaw-UK set carries four things: the question, a gold answer, the citations a correct answer must include, and the keywords the grader checks for. Here is a real one from the curated tier:
- Question: "Which FCA principle introduces the Consumer Duty?"
- Gold answer: the Consumer Duty is Principle 12, requiring firms to act to deliver good outcomes for retail customers.
- Required citation: PRIN 12.
- Graded keywords: Consumer Duty, good outcomes, Principle 12.
Composition matters more than size
The full set is 110 items, and the split matters more than the count: 80 short-form questions, 20 document-level tasks, and 10 case-like scenarios, spanning domains from FSMA authorisation and conduct-of-business rules to payment services and complaints handling.
Each tier probes a different failure mode. Short-form questions check retrieval precision: can the system find the one rule that answers a pointed question? Document tasks check synthesis: can it hold a whole chapter in view without dropping the cross-references? Scenarios check application: given a fact pattern, does it reach for the right regime at all? A benchmark that is all short-form questions will happily certify a system that falls over the moment a question spans two documents.
The aggregate score hid the story
The headline numbers looked healthy: 0.82 source accuracy, 0.81 citation quality, 0.76 RAGAS faithfulness, 0.74 answer relevance. Source accuracy peaked at 0.85 on advanced queries, and document tasks cleared 0.83 on both source accuracy and citation quality - exactly the cases where graph expansion supplies the regulatory hierarchy that dense retrieval drops.
An aggregate score is an average of your best and worst subsystems pretending to be one number.
Where it actually failed
The per-domain breakdown told a different story in one corner: consumer redress. DISP - the complaints-handling sourcebook - dragged legal completeness below 0.65. The reason is structural. DISP rules are fragmented into many small, heavily cross-referenced fragments, and the one-hop graph expansion that stitches context back together for the rest of the Handbook could not reassemble them. Rule-level chunking is the right default for most of the corpus; for DISP it shreds the very context the expansion is supposed to recover.
That is a finding I only have because the benchmark was built to expose it: domain labels on every item meant the failure had an address, not just a lower average.
What I would build next
Two things. Domain-aware chunk granularity - or a second expansion hop - for fragmented sourcebooks like DISP, so completeness recovers where the structure is hostile. And a structured legal-reasoning benchmark beyond RAGAS, because faithfulness scoring cannot catch an answer that is perfectly grounded in a retrieved passage and still misapplies it across jurisdictions. The eval set is public in the repo alongside the pipeline; if you are building anything similar, steal the format - question, gold answer, required citations, graded keywords. The fourth column is the one you will thank yourself for.