LLMs Waffle and Err on GOV.UK Queries

• LLMs tested included both large closed-source models (ChatGPT 4.1) and smaller open-source alternatives (Llama 3.1 8B, Qwen3-32B, ChatGPT-OSS-20B)[3] • Models were evaluated on three dimensions: verbosity, accuracy, and refusal rates across 22,000+ government service queries[3] • Error patterns are inconsistent and unpredictable rather than systematic, suggesting fundamental limitations in how LLMs process and validate factual information[3] • The research indicates that increasing model size or prompt engineering alone will not significantly improve safety in high-stakes domains like government services[5] • Proposed technical solutions include 'model immunisation'—fine-tuning models on curated sets of explicitly labeled falsehoods to build resistance to misinformation[5]

The ODI findings raise critical questions about LLM deployment in public-facing government services and official information dissemination. As the UK develops AI tutoring tools for secondary schools (trials beginning summer 2026)[4] and implements its principles-based AI regulatory framework, this research demonstrates that current LLMs cannot reliably serve as authoritative sources for official information without substantial safeguards. The inconsistency of errors suggests that organizations cannot rely on simple mitigation strategies; instead, they must implement context-sensitive safeguards and human oversight for any LLM application involving government services, legal information, or public welfare. The finding that smaller models perform comparably to larger ones may accelerate adoption of cost-effective alternatives, but only if accompanied by rigorous validation protocols. This research will likely influence the comprehensive AI Bill expected in 2026 and shape how UK regulators approach LLM governance in regulated sectors.