FinLaw-UK

The pipeline begins with bulk ingestion of FCA Handbook chapters and adjacent regulatory documents. Each section is chunked at the smallest semantic unit - typically a single rule or sub-rule - then run through a parallel two-stream extraction: Sentence Transformer embeddings into a vector index, and entity/relationship extraction into a Neo4j knowledge graph that captures Rule → Chapter, Rule → Entity, and Rule → Cross-reference relationships.

At query time retrieval is hybrid: BM25 lexical search and BGE-small dense search run in parallel and are fused by reciprocal rank fusion, so an exact rule reference and a paraphrased description both land. The fused candidates' graph nodes are expanded two hops in Neo4j to add adjacent rules, parent chapter context, and cross-referenced sections - the structural context pure vector search loses. A cross-encoder re-ranks the expanded set before it is trimmed to fit Mistral 7B-Instruct's context window.

Generation runs locally on Mistral 7B-Instruct with strict citation-required prompting: every claim must reference a chunk ID, and every chunk ID must exist in the retrieved set. Post-generation, every response goes through a RAGAS evaluator that scores faithfulness (does the answer stay grounded in retrieved context?) and answer relevance (does it actually address the query?).

Evaluation runs against a 110-item benchmark I built and released: 80 factual questions, 20 document tasks, 10 case scenarios. The pipeline reaches 0.76 RAGAS faithfulness and 0.74 answer relevance, with a custom legal-completeness metric at 0.68.

The result I would lead with is a correction to my own submitted work. Re-measuring the evaluation after submission showed that the reported source-accuracy and citation-quality figures were regex shape-checks, not correctness measures: the scorer awarded a flat 0.85 to any citation-shaped string, and 103 of the 110 rows carried that constant. The true graph-verified citation rate was 3 in 110. Both figures are withdrawn, and the repository ships a measurement-integrity report stating which numbers reproduce and which do not.

The figures that do reproduce: 0.76 RAGAS faithfulness, 0.74 answer relevance, and a custom legal-completeness metric that holds at 0.68 across both evaluation tracks. That correction is what motivated the citation normaliser, the cross-encoder re-ranker and the refusal gate.

Refusing costs something measurable, and it is worth paying. Headline answer relevancy drops 23 points because RAGAS scores a refusal as zero by construction. Exclude the 30 refusals and the mean is 0.658, against 0.641 for the non-refusing baseline - which scored higher only by confabulating answers to questions the corpus cannot support.

The measurement bug is the part I would want a supervisor to press me on. A scorer returning the same constant for 103 of 110 rows is visible in the distribution and invisible in the mean, and I only caught it because I went back to reproduce my own numbers after the dissertation was already submitted. Publishing the correction next to the code was the only defensible option, and it changed what the system does: the refusal gate exists because the citation numbers did not survive scrutiny.

If I were continuing this past the dissertation, the next move is two-pronged. First, extend the evaluation harness into a structured legal-reasoning benchmark - RAGAS catches faithfulness drift but not legal-specific failure modes like jurisdictional misapplication. Second, ship a confidence-aware UI that surfaces uncertainty when the graph expansion returns sparse adjacency, so users know when the system is reasoning from rich vs thin context.