Agentic Tool-Loop Thrashing in Multi-Jurisdiction Legal RAG

Measured, audited system

Pass 1 returned £68,400 (correct). Passes 2 and 3 returned £72,300 (wrong by £3,900). 186-second unconstrained trace, 117 fragments retrieved. 7 August 2026, question and answer fixed before the run.

Measured, reference implementation

£68,400 (SI 2010/2926) on 1,000 of 1,000 passes. p99 0.19 ms. 0 LLM or tool calls. Same test question, 17 October 2026.

By design

The agent orchestrator stops multi-hop search once a citation is bound.

Retrieval Integrity Under Statute Collision in Legal RAG

Measured, audited system

Wrong statute returned on 3 of 3 article-collision probes against Spanish statutory law. Black-box, April 2026.

Measured, reference implementation

UK legislation: 0 of 40 same-numbered sections bound to the wrong Act, 0 of 1,030 unseen real citations bound wrongly, 0 of 68 invented Acts bound. Test set fixed before the run, 29 September 2026.

By design

Deterministic statute pre-filtering binds the target code before vector embedding, so retrieval draws from that statute only.

Generation-Tier Integrity in Legal RAG via In-Flight Verification

Measured, reference implementation

Rollback retry 117 ms against 270 ms to re-process, one NVIDIA L4. With an LLM judge, closed-book wrong answers fell from 198 to 28 of 240. UK statutes, October 2026.

By design

Rejected sentences leave the generation state; later tokens condition only on verified text. Per-query audit log, interface labelling and monitoring.

Confidential Legal AI Without Trusted Hardware

Measured, reference implementation

Key-based model hiding: 100% of secret content recovered. Laptop-only design: 0 of 17 secrets reached the cloud; cloud RAG leaked 16 private tokens. 30 of 30 conclusions correct on unseen matters. Task routing failed its rule, 7 of 8. Fictional matters, October 2026.

By design

Cloud compute on client data needs a hardware trusted execution environment (TEE) with attestation.

Can You Build Confidential Legal AI Without a TEE? Empirical Lessons on PIR and Keyed Models

Two studies on fictional UK employment matters, October 2026, with the cloud provider treated as an attacker with full access. A model hidden behind a secret key was read in full: 100% of the secret content recovered from public statistics of its weights. Keeping client work on a laptop and fetching public law by private information retrieval sent 0 of 17 client secrets to the cloud, where a standard cloud pipeline sent 16 private tokens. With dates and amounts computed in code, 30 of 30 conclusions on 8 unseen matters were correct. Work that needs cloud compute on client data needs a hardware trusted execution environment, such as Prem Enclave.

In-Flight Rollback on SGLang: A Sentence-Level Check Cut Wrong Legal Answers From Memory by 83 to 86%

legal-rag-verifier checks each sentence while the model writes it, cuts a rejected one and rewrites it with the statutory text added. 240 UK statute questions answered from memory, 14 October 2026: answers holding a wrong sentence fell from 132 to 23 for Qwen 27B and from 198 to 28 for Qwen 2.5 7B. After a real error, the default retry was wrong again 71 times in 95, and a retry with the provision added at the cut 16 times. With the source already in the prompt, Qwen 27B had 4 such answers with the loop and 4 without, against 3 for Gemini 3.8 Flash and 8 for Claude Opus 4.6.

What Failed in My Own Controls: Seven Results From October 2026

Seven failures in my own legal AI controls, five measured against a rule set before the run and two found in review, each with its date, configuration and evidence: an auditor whose rejections were mostly wrong, a model-hiding scheme recovered 100%, routing that failed its pre-registered rule twice, two missed latency targets, a calculator fallback and a wrong statutory instrument on my own case study.

legal-rag-router: Deterministic Citation Routing for UK Legal RAG, Measured Against Gemini, Jev and Laya

An open-source router that binds a legal RAG query to the cited UK provision before vector search. Blind sealed run, 29 September 2026: 0 of 1,897 real citations bound to the wrong instrument. Gemini 3.8 Flash (high) routing on its own, 1 October 2026: 102 of 1,897.

Jev System One vs Specialized Sovereign RAG. An Empirical Benchmark

Across 20 measured trials on Apple Silicon MPS, a cloud-hosted decision primitive averaged 457.19ms P50 (381.0ms upstream cloud floor) and consumed 453 tokens before retrieval began. A co-hosted sovereign classifier completed the same routing in 0.23ms with zero tokens.