FinLaw-UK was my MSc dissertation: a retrieval-augmented generation pipeline for UK financial regulation. The corpus is genuinely hard. The FCA Handbook alone runs to thousands of pages, cross-referenced with MiFID II, the PRA Rulebook, and a stack of binding technical standards. A naive LLM lookup answers confidently and wrongly in exactly the places where being wrong is most expensive.
Building the retrieval and generation was the interesting engineering. The harder question came afterwards: how do I know an answer is actually grounded in the source, and not just fluent? That question is where RAGAS earned its place in the project.
Fluent and correct are not the same thing
The first thing to accept is that surface metrics do not work here.
BLEU and ROUGE reward text that reads like the reference. For regulatory text, that is close to worthless - a fluent paraphrase with one wrong citation is a liability, and it scores just as well as a correct answer. What I needed to measure was different: does the answer stay grounded in the context that was actually retrieved, and does it address the question that was actually asked? RAGAS scores exactly those two things, faithfulness and answer relevance, and both need an LLM judge reading the answer against the retrieved context rather than against a golden string.
A polished answer and a hallucinated one look identical until you check them against the source.
What faithfulness actually checks
Faithfulness asks whether every claim in the answer is supported by the retrieved context. The idea is easier to see as a sketch than to describe:
# Illustrative sketch of the idea - RAGAS uses an LLM judge, not this code
claims = extract_claims(answer)
supported = [c for c in claims if entailed_by(c, retrieved_context)]
faithfulness = len(supported) / max(len(claims), 1)A score near 1.0 means the answer invented nothing; a lower score means part of it came from the model's parameters rather than the documents in front of it. That framing quietly moves the burden onto retrieval - the generator can only be as faithful as the context it is handed.
The retrieval has to earn the score
This is where plain vector search falls down. Dense retrieval finds passages that are semantically similar, but it misses the structure: a single rule is only meaningful alongside its parent chapter, the entities it obligates, and the rules it cross-references. Similarity does not recover that. The regulatory graph does.
So ingestion runs two streams in parallel. Each section is chunked to its smallest semantic unit - usually a single rule - then embedded with Sentence Transformers into a vector index and, separately, parsed into a Neo4j knowledge graph that captures Rule-to-Chapter, Rule-to-Entity, and Rule-to-Cross-reference relationships.
Generation itself is the easy part to describe: it runs locally on Mistral 7B-Instruct under citation-required prompting, where every claim has to reference a chunk ID and every chunk ID it cites has to exist in the retrieved set. That constraint is what makes a faithfulness score meaningful rather than aspirational. The interesting work happens before it, at query time:
- Dense retrieval pulls the top-K candidate chunks.
- A single-hop Cypher expansion in Neo4j adds adjacent rules, parent-chapter context, and cross-referenced sections.
- The expanded set is re-ranked and trimmed to fit Mistral 7B-Instruct's context window.
The decisions that shaped it
A few of them I would defend again:
- Graph expansion over a better reranker - the failure mode was missing context, not mis-ranked context, and no reranker recovers a passage that was never retrieved.
- Neo4j over a vector-only store like Pinecone or Qdrant - the cross-reference graph is the actual moat, and one Cypher hop expands it cheaply.
- Mistral 7B over a frontier API - a small open-weight model running locally proved the retrieval gains held regardless of generator size, with no vendor dependency or data-egress problem on sensitive text.
What the numbers said, and what they didn't
On a 110-item benchmark spanning seven regulatory domains, the pipeline reached 0.76 faithfulness and 0.74 answer relevance, with legal completeness at 0.68. Two other figures were in this paragraph until I re-measured the evaluation after submitting it: source accuracy and citation quality both came from a scorer that checked whether a string looked like a citation, not whether it was the right one. Both are withdrawn, and the correction is in the repo.
The honest reading of 0.76 is that roughly three in four answers stay grounded and about a quarter drift. Those are not numbers to admire; they are the prompt-engineering targets for the next iteration. Answer relevance at 0.74 tracks faithfulness closely, which suggests the model stays on topic when it stays grounded - the two failure modes look related, not independent.
Where RAGAS stops helping
RAGAS is good at catching faithfulness drift. It is not good at catching legal-specific failure modes. An answer can be perfectly grounded in a retrieved passage and still misapply it across jurisdictions, and faithfulness will score that highly. If I were taking this past the dissertation, the next moves are to extend the evaluation harness into a structured legal-reasoning benchmark, and to ship a confidence-aware interface that surfaces uncertainty when the graph expansion returns thin adjacency - so the person reading the answer knows whether it was reasoned from rich context or from sparse.