legal-rag-router: Deterministic Citation Routing for UK Legal RAG, Measured Against Gemini, Jev and Laya
legal-rag-router is an open-source Python library. It reads the legal citation in a query, checks it against an index of the UK statute book, and returns one decision before any vector search runs. The decision is a coordinate to retrieve from, a question for the user, or a refusal.
Version 0.1.0 covers UK legislation. It is on GitHub and PyPI, with a summary on /tools.
Summary
All figures are on 2,112 labelled queries, index snapshot 28 September 2026.
- Measured, blind sealed run, 29 September 2026. 0 of 1,897 real citations bound to the wrong instrument. 0 of 68 invented Acts bound. 82 of 1,030 held-out real citations refused.
- Measured, 1 October 2026. Gemini 3.8 Flash at the high thinking level, routing on its own, bound the wrong instrument for 102 of 1,897. Used as a citation parser in front of the router's index, it was the most accurate configuration measured, with 0 wrong bindings.
- Measured, 30 September 2026. Router p50 0.14 ms, p99 2.10 ms on an Apple M4. The p99 target of 2 ms is missed.
The problem
Vector search returns the text nearest to the query. It does not return the provision the query cites. Asked about an Act that does not exist, it still returns something.
Two measurements on this site show the result.
- Measured. Spanish statutory law, April 2026, black-box probes against a live public interface. 3 of 3 article-collision probes returned the wrong statute. Three probes show reproducibility, not a rate. Case study.
- Measured. A commercial multi-jurisdiction system, 7 August 2026, one UK query run three times. The agent made 15 or more tool calls over 186 seconds, retrieved 117 fragments, 57 of them French labour code articles, and returned £68,400 once and £72,300 twice. £68,400 was correct. Case study.
In both cases the query named the law. The failure was in what retrieval was allowed to search.
The router addresses this class of failure for UK legislation only. It does not cover Spanish law, and it has not been run against either audited system.
What it does
One call returns one result. The result carries a status and a next action for the pipeline.
"What is the cap in section 124 of the Employment Rights Act 1996?"
ROUTE_BOUNDED uk/ukpga/1996/18/s124 retrieve only from here
"section 3 of the Digital Privacy Rights Act 2021"
EPISTEMIC_ABSTENTION_INSTRUMENT_NOT_FOUND refuse: no such Act
| Status | Next action | When |
|---|---|---|
ROUTE_BOUNDED |
Retrieve, filtered to the coordinate | The citation resolves |
ROUTE_AMBIGUOUS |
Ask the user | "the Companies Act" matches several Acts |
EPISTEMIC_ABSTENTION_INSTRUMENT_NOT_FOUND |
Refuse | The Act or SI is not in the statute book |
EPISTEMIC_ABSTENTION_PROVISION_NOT_FOUND |
Refuse | The Act exists, the section does not |
ROUTE_OUT_OF_COVERAGE |
Declare out of coverage | Scottish, Welsh, NI, EU or retained EU law |
ROUTE_UNRESOLVED |
Discover, then bind | The query cites nothing |
A bound result produces a partition filter. The filter restricts vector search to the bound instrument and provision. For any other status the filter function raises an error, so there is no unbounded filter to fall back to.
A query that cites nothing goes to discovery. Discovery offers candidate provisions for the user to confirm. It never binds on its own.
By design, these properties follow from the architecture:
- No model, no network call, no disk read per query.
- The same input returns the same output.
- Zero runtime dependencies. Standard library only.
- Every index file is checked against a SHA-256 hash at load. A damaged index fails at load, not at query time.
- The index holds coordinates, titles and lookup tables. It holds no provision text. Your store keeps the text.
Architecture
The router runs inside your process. Every query takes one pass through the steps below. A step that cannot complete safely returns ROUTE_UNRESOLVED. No step guesses.
flowchart TD
Q["Query text"]
subgraph ROUTER ["legal-rag-router: in process, no model, no network "]
PRE{"Precheck<br/>text, under 4 KB, UK in scope"}
SIG{"Citation signal?<br/>normalise, prefilter"}
SCAN["Scan<br/>titles, abbreviations, chapter and SI numbers,<br/>pinpoints, ranges, exclusions, date cues"]
CAP{"Within size and<br/>complexity limits?"}
LINK["Link provisions to instruments<br/>apply exclusions and prior-turn context"]
RES["Resolve each citation against the index<br/>typo tiers: correct, ask or refuse"]
IDX[("Index, snapshot 28 Sep 2026<br/>134,219 instruments, coordinates and lookup tables<br/>memory-mapped, SHA-256 verified at load")]
OUT["Assemble one result<br/>status, next action, coordinates,<br/>corrections, temporal hint, repealed flag"]
PRE -->|pass| SIG
SIG -->|yes| SCAN
SCAN --> CAP
CAP -->|yes| LINK
LINK --> RES
IDX -.-> RES
RES --> OUT
end
BOUND["ROUTE_BOUNDED"]
AMB["ROUTE_AMBIGUOUS"]
NF["INSTRUMENT_NOT_FOUND<br/>PROVISION_NOT_FOUND"]
OOC["ROUTE_OUT_OF_COVERAGE"]
UNR["ROUTE_UNRESOLVED"]
subgraph PIPE ["Your pipeline"]
FILTER["partition_filter<br/>instrument_id and coordinate"]
VS[("Your vector store<br/>holds the provision text")]
GEN["Generation"]
ASK["Ask the user<br/>candidates and a clarification question"]
REF["Refuse<br/>states the snapshot date"]
DECL["Declare out of coverage"]
DISC["discover<br/>concept index offers candidate provisions"]
CONF["User confirms one<br/>routed again as a new query"]
FILTER --> VS
VS --> GEN
DISC --> CONF
end
Q --> PRE
PRE -->|fail| UNR
SIG -->|no| UNR
CAP -->|no| UNR
OUT --> BOUND
OUT --> AMB
OUT --> NF
OUT --> OOC
OUT --> UNR
BOUND --> FILTER
AMB --> ASK
NF --> REF
OOC --> DECL
UNR --> DISC
By design:
- The only path to generation runs through
ROUTE_BOUNDEDand the partition filter. The filter function raises an error for every other status. - Every chunk in your vector store carries
instrument_idandcoordinate. The filter matches on those two fields. - Limits on query length, citation count and lookups keep routing time linear in the query length. A query past a limit returns
ROUTE_UNRESOLVED.
Coverage
- 134,219 UK Public General Acts and Statutory Instruments, bound to provision level.
- Index snapshot 28 September 2026. Every refusal states that date.
- Citation forms: full and short titles, abbreviations, chapter and SI numbers, pinpoints, ranges, exclusions, typos.
- Recognised and reported out of coverage: Scottish, Welsh and Northern Ireland legislation, local Acts, EU and retained EU law.
- Not covered in 0.1.0: Spanish law. It is on the roadmap.
How it was measured
The test batteries were hashed and tagged before the router ran on them. The router ran once. The results were hashed after. The seals are in the repository.
- 2,112 labelled queries across nine batteries.
- 1,030 of them are real citations taken from legislation text and held out from development.
- v1 is the blind run, 29 September 2026. v2 reran after fixes to failures v1 exposed, so v2 is not blind on the hand batteries. Both are published. This article uses v1 unless it says otherwise.
Results
Measured. Sealed run v1 (blind), 29 September 2026, 2,112 queries, index snapshot 28 September 2026.
| Measure | Result |
|---|---|
| Real citations bound to the wrong instrument | 0 / 1,897 (95 % upper bound 0.16 %) |
| Held-out real citations bound wrongly | 0 / 1,030 |
| Invented Acts bound | 0 / 68 |
| Same section number in different Acts, bound to the wrong Act | 0 / 40 |
| Real law refused | 84 / 1,897 (v2: 82) |
| Rows with the expected outcome | 1,903 / 2,112 (v2: 1,916) |
Measured. Latency, 30 September 2026, full UK index, 306,336 calls per pass, 3 passes.
| Machine | p50 | p99 |
|---|---|---|
| Apple M4 | 0.14 ms | 2.10 ms |
| Intel i5-3570 (2012) | 0.50 ms | 9.21 ms |
Measured. 100,000 real citations taken from the text of UK legislation, index snapshot 28 September 2026. These citations were used to find and fix errors during development, so the router was partly tuned on them. The held-out results above use a separate set it never saw.
| Outcome | Share |
|---|---|
| Bound correctly | 80.02 % |
| Missed | 8.54 % |
| Refused | 6.33 % |
| Asked about | 4.37 % |
| Bound to another instrument | 7 citations |
How it compares
Measured. Run 1 October 2026, scored 2 October 2026, with the sealed run's own scoring code. Fed the router's answers, that code reproduces the v2 run exactly.
- Gemini 3.8 Flash at the high thinking level, through Google AI Studio. It was given the router's output contract, coverage, snapshot date and nine worked examples that do not overlap the battery. Its answer was read two ways. As a parser, it lists the citations and the router's index resolves them. As a router, it returns the outcome and coordinates on its own, from memory.
- Jev (
typesafe/jev-1.13-20260917), through OpenRouter's Decisions API. Jev is a decision model, so it was asked choice questions only. - Laya (421M base checkpoint), self-hosted. Speed and footprint only. Its accuracy is not reported, for the reason under Choice questions.
Routing the battery
Same 2,112 queries for every column. Router v1 is the blind run. Router v2 is the code that ships as 0.1.0, rerun after fixes to v1's failures, so it is not blind on the hand-written batteries. Gemini was run on the v2 battery and had not seen it.
| Measure | Router v1 (blind) | Router v2 (0.1.0) | Gemini parses, router resolves | Gemini alone |
|---|---|---|---|---|
| Real citations bound to the wrong instrument | 0 / 1,897 | 0 / 1,897 | 0 / 1,897 | 102 / 1,897 (5.38 %) |
| Real law refused | 84 / 1,897 | 82 / 1,897 | 55 / 1,897 | 233 / 1,897 |
| Held-out real citations left unresolved | 67 / 1,000 | 67 / 1,000 | 11 / 1,000 | 11 / 1,000 |
| Invented law bound | 0 / 68 | 0 / 68 | 0 / 68 | 0 / 68 |
| Ambiguity battery, expected outcome | 23 / 25 | 25 / 25 | 8 / 25 | 2 / 25 |
| Typos bound that needed a question or refusal | 1 / 22 | 0 / 22 | 4 / 22 | 3 / 22 |
| Rows with the expected outcome | 1,903 / 2,112 | 1,916 / 2,112 | 1,972 / 2,112 | 1,799 / 2,112 |
- Gemini on its own bound the wrong instrument for about 1 in 19 real citations and refused about 1 in 8.
- Gemini as a parser in front of the router was the most accurate configuration measured. It resolved 116 real-document rows the router left unresolved or refused, and lost 8 the router got right. Most were bare SI numbers, multi-citation sentences and schedule paragraphs.
- That configuration bound nothing wrong and nothing invented, because every coordinate came from the router's index.
- On ambiguous citations the router asked. Gemini picked one instrument.
- Gemini refused all 68 invented citations, given a contract that told it how.
Choice questions
The right instrument is always among the options, each labelled with its title and citation, plus "none of these". This is the ceiling with a retriever that never misses. 2,050 questions per option count, instrument level only.
| Options | Jev correct | Gemini correct |
|---|---|---|
| 3 | 2,031 / 2,050 (99.1 %) | 2,037 / 2,050 (99.4 %) |
| 10 | 2,033 / 2,050 (99.2 %) | 2,038 / 2,050 (99.4 %) |
| 30 | 2,023 / 2,050 (98.7 %) | 2,035 / 2,050 (99.3 %) |
Laya is not in this table. Its vendor documents the base checkpoint as near chance without fine-tuning: 0.362 on the vendor's own typed-decisions benchmark, against 0.727 for Jev. Its only fine-tuned checkpoint is built for four synthetic workflows outside law, and its model card says to expect base-level results or worse elsewhere. No legal fine-tune of Laya exists. Scoring the base checkpoint would measure a use its vendor does not support. A legal fine-tune is future work, and it would need training data kept apart from the sealed batteries.
Both models choose well when the right instrument is offered. The open question is getting it into the options. Laya, at 30 options, cuts each title to 5 tokens and sees a median 29 % of it. "The Welfare Reform Act 2012 (Commencement No. 9, 21 and 23 (Amendment)) Order 2018" reaches the model as "The Welfare Reform Act". It raises no warning.
Speed
Measured. Time per call. The router and Laya were measured on 30 September 2026, Jev and Gemini on 1 October 2026. Laya ranges cover questions with 3, 10 and 30 options.
| System | Where it ran | Calls | p50 | p99 |
|---|---|---|---|---|
| Router | Apple M4, in process | 306,336 × 3 | 0.14 ms | 2.10 ms |
| Router | Intel i5-3570 (2012), in process | 306,336 × 3 | 0.50 ms | 9.21 ms |
| Router | Modal, 1 CPU core, us-east, time inside the container | 6,336 | 0.63 ms | 3.65 ms |
| Router | Same Modal service, round trip from Pakistan | 6,336 | 276.6 ms | 364.5 ms |
| Laya | NVIDIA T4 GPU on Modal | 900 | 34 to 37 ms | 44 to 46 ms |
| Laya | Apple M4 GPU | 900 | 50 to 81 ms | 88 to 117 ms |
| Laya | Apple M4 CPU | 900 | 89 to 139 ms | 137 to 170 ms |
| Laya | Intel i5-3570 CPU | 900 | 1,165 to 1,987 ms | 2,336 to 2,706 ms |
| Jev | OpenRouter, from Pakistan | 287 | 507 ms | 2,222 ms |
| Gemini 3.8 Flash (high) | Google AI Studio, from Pakistan | 300 | 13.5 s | 192 s |
- Router on Modal. The round trip is almost all network. A call with no routing on the same connection took 275.7 ms at p50. The router's answers matched the local router on all 6,336 calls.
- Router inside the hosted-call pass. With caches cold after each network wait, its p50 on the M4 was 1.18 ms.
- Laya. On its vendor's reference GPU, the T4, it took 34 to 37 ms. The vendor publishes 33 to 40 ms. Without a GPU it is 89 ms or more per call.
- Jev. A call with no model on the same connection took 103 ms at p50.
- Gemini. The time is the model's own. 22 of 300 calls took over a minute. The slowest took 249 s and 32,497 thinking tokens, on a dense amendment-note citation.
Cost, repeatability, footprint, data
| Router | Laya | Jev | Gemini 3.8 Flash (high) | |
|---|---|---|---|---|
| Answer changed across 5 repeats, 200 rows | 0 | Not measured | 2 / 197 | 34 / 200 |
| Cost per 1,000 queries, list price | $0 | GPU time | $0.041 | $8.76 |
| Install, plus model or index | 0.6 MB, plus 362 MB index | 693 MB, plus 807 MB weights | Hosted | Hosted |
| Query text leaves your network | No | No | Yes | Yes |
Gemini's cost is the promotional price valid to 31 December 2026. The regular price is $17.52 per 1,000.
What was not controlled
- One run, at one time of day, from one client in Pakistan. No off-peak run and no UK or EU client.
- Gemini's figures hold for the high thinking level only.
- Choice-question results are instrument level only, with the right answer always offered.
- Run cost: Gemini $37.78 at the promotional price, Jev $0.30.
By design, not measured. Call the LLM only on queries the router leaves unresolved or refuses. In this battery that is 258 of 2,112 queries, 12 %. The router's ROUTE_UNRESOLVED status already routes to that step.
Every call, the seals and the scoring code are in the comparison report.
Where it fails
- Real law is refused. The router refused 82 of the 1,030 held-out real citations. Most come from amendment notes in legislation and cite a section that has since been repealed, so it is no longer in the current statute book. The router refuses a provision it cannot find in the current text instead of binding the nearest one. That is a design decision. A question about the law before the repeal needs a point-in-time source.
- Citations are missed. 8.54 % of the 100,000 development citations. The commonest missed shape is a bare SI number such as
1999/3434, which the router does not bind on purpose. - Some citations bind to the wrong instrument. 7 of the 100,000 development citations. None of the 1,030 held-out citations.
- The latency target is missed. The design target was p99 under 2 ms. Measured p99 is 2.10 ms on the M4 and 9.21 ms on the i5. The tail is refusals, which rank suggested titles.
- "As at" dates are not read. Verified in isolation, 2 October 2026: "As at 1 January 2012" binds the right provision but returns no
temporal_hint. "As it stood on" and "as in force on" are read. "As at" and "as of" are planned for the next release. - Discovery is limited. Measured, sealed test run, 29 September 2026, 91 drafted queries: the right provision is in the top 10 for 88 % and first for 51 %. For the colloquial query "What is the cap on unfair dismissal?" the top 3 did not include section 124 (Verified in isolation, 2 October 2026).
- The index is a snapshot. Law made after 28 September 2026 is refused until the index is rebuilt. There is no live register check in 0.1.0.
What it does not do
- It does not hold or return provision text.
- It does not resolve a date to the version in force. It passes the date through for a point-in-time layer.
- It does not check what the model writes. A bound coordinate limits where retrieval looks. It does not establish that the answer is correct.
Install
pip install legal-rag-router
Python 3.11 or later. The index is a separate download attached to each GitHub release, with a SHA256SUMS file. Licence: AGPL-3.0-only. Index data is derived from legislation.gov.uk under the Open Government Licence v3.0.
Source code, documentation, the sealed batteries and every result file: github.com/azterizm/legal-rag-router.
Run this measurement against your own system.
What this costsMore Articles You Might Like
The EU AI Act & Legal RAG: Why 'Approve' Buttons Fail Compliance Audits and How to Build Deterministic Human Oversight
From Article 14 Human Oversight to Entity Masking Engines: Architectural blueprints for engineering auditable, deterministic legal AI under strict EU/UK regulatory regimes.
Jev System One vs Specialized Sovereign RAG. An Empirical Benchmark
Across 20 measured trials on Apple Silicon MPS, a cloud-hosted decision primitive averaged 457.19ms P50 (381.0ms upstream cloud floor) and consumed 453 tokens before retrieval began. A co-hosted sovereign classifier completed the same routing in 0.23ms with zero tokens.
The 51.4% Problem: Why Post-Hoc Validation Fails Legal RAG Governance
Why post-hoc validation causes a 51.4% abstention rate in legal RAG governance, and how in-flight sentence-level NLI verification solves the hallucination tax.
New entries are published to the feed.
RSS — /feed.xml