Agentic Tool-Loop Thrashing in Multi-Jurisdiction Legal RAG

Measured, audited system

Pass 1 returned £68,400 (correct). Passes 2 and 3 returned £72,300 (wrong by £3,900). 186-second unconstrained trace, 117 fragments retrieved. 7 August 2026, question and answer fixed before the run.

Measured, reference implementation

£68,400 (SI 2010/2926) on 1,000 of 1,000 passes. p99 0.19 ms. 0 LLM or tool calls. Same test question, 17 October 2026.

By design

The agent orchestrator stops multi-hop search once a citation is bound.

Retrieval Integrity Under Statute Collision in Legal RAG

Measured, audited system

Wrong statute returned on 3 of 3 article-collision probes against Spanish statutory law. Black-box, April 2026.

Measured, reference implementation

UK legislation: 0 of 40 same-numbered sections bound to the wrong Act, 0 of 1,030 unseen real citations bound wrongly, 0 of 68 invented Acts bound. Test set fixed before the run, 29 September 2026.

By design

Deterministic statute pre-filtering binds the target code before vector embedding, so retrieval draws from that statute only.

Generation-Tier Integrity in Legal RAG via In-Flight Verification

Measured, reference implementation

Rollback retry 117 ms against 270 ms to re-process, one NVIDIA L4. With an LLM judge, closed-book wrong answers fell from 198 to 28 of 240. UK statutes, October 2026.

By design

Rejected sentences leave the generation state; later tokens condition only on verified text. Per-query audit log, interface labelling and monitoring.

Confidential Legal AI Without Trusted Hardware

Measured, reference implementation

Key-based model hiding: 100% of secret content recovered. Laptop-only design: 0 of 17 secrets reached the cloud; cloud RAG leaked 16 private tokens. 30 of 30 conclusions correct on unseen matters. Task routing failed its rule, 7 of 8. Fictional matters, October 2026.

By design

Cloud compute on client data needs a hardware trusted execution environment (TEE) with attestation.

In-Flight Rollback on SGLang: A Sentence-Level Check Cut Wrong Legal Answers From Memory by 83 to 86%

legal-rag-verifier checks each sentence while the model writes it, cuts a rejected one and rewrites it with the statutory text added. 240 UK statute questions answered from memory, 14 October 2026: answers holding a wrong sentence fell from 132 to 23 for Qwen 27B and from 198 to 28 for Qwen 2.5 7B. After a real error, the default retry was wrong again 71 times in 95, and a retry with the provision added at the cut 16 times. With the source already in the prompt, Qwen 27B had 4 such answers with the loop and 4 without, against 3 for Gemini 3.8 Flash and 8 for Claude Opus 4.6.

What Failed in My Own Controls: Seven Results From October 2026

Seven failures in my own legal AI controls, five measured against a rule set before the run and two found in review, each with its date, configuration and evidence: an auditor whose rejections were mostly wrong, a model-hiding scheme recovered 100%, routing that failed its pre-registered rule twice, two missed latency targets, a calculator fallback and a wrong statutory instrument on my own case study.

legal-rag-router: Deterministic Citation Routing for UK Legal RAG, Measured Against Gemini, Jev and Laya

An open-source router that binds a legal RAG query to the cited UK provision before vector search. Blind sealed run, 29 September 2026: 0 of 1,897 real citations bound to the wrong instrument. Gemini 3.8 Flash (high) routing on its own, 1 October 2026: 102 of 1,897.

Jev System One vs Specialized Sovereign RAG. An Empirical Benchmark

Across 20 measured trials on Apple Silicon MPS, a cloud-hosted decision primitive averaged 457.19ms P50 (381.0ms upstream cloud floor) and consumed 453 tokens before retrieval began. A co-hosted sovereign classifier completed the same routing in 0.23ms with zero tokens.

Why Telling an LLM Not to Look Guarantees It Will

Vals AI's BioMysteryBench audit confirms what the attention mechanism already predicts: instructing a model to ignore data forces it to attend to that data. The fix is physical, not linguistic.