๐ArXiv AIโขStalecollected in 13h
EHRBench: Large-Scale Benchmark for Clinical Decision Making
๐กA massive, reliable 1M-item benchmark for evaluating LLM clinical reasoning using real-world medical data.
โก 30-Second TL;DR
What Changed
Constructed nearly 1M QA items from real patient EHR trajectories.
Why It Matters
This benchmark provides a critical tool for researchers to quantify the reliability of LLMs in high-stakes medical environments. It helps bridge the gap between general language capabilities and the specific inference requirements of clinical practice.
What To Do Next
If you are developing medical AI, use EHRBench to stress-test your model's clinical reasoning against its 1M-item dataset to identify specific hallucination patterns.
Who should care:Researchers & Academics
Key Points
- โขConstructed nearly 1M QA items from real patient EHR trajectories.
- โขUtilizes an EHR-LLM-KB interaction pipeline to ensure data reliability and reduce hallucinations.
- โขEvaluates 30+ LLMs across three core clinical tasks: diagnosis, treatment, and prognosis.
- โขProvides a standardized framework to measure clinical inference capabilities in medical AI.
๐ง Deep Insight
Web-grounded analysis with 10 cited sources.
๐ Enhanced Key Takeaways
- โขEHRBench addresses the scalability limitations of previous benchmarks, which were often restricted by the high cost and manual effort of curation, by offering an automated construction pipeline for large-scale evaluation.
- โขThe benchmark's construction involves a specialized Large Language Model (LLM) to automatically convert encounter-level Electronic Health Record (EHR) trajectories into structured templates, which are then deterministically instantiated into nearly 1 million question-answering (QA) items.
- โขSystematic knowledge-base (KB) based verification and enrichment are integral to the EHR-LLM-KB interaction pipeline, specifically designed to filter out hallucinated or ambiguous relations and enhance the overall reliability of the generated data.
- โขThe evaluation process for EHRBench is implemented in Python 3.11.5 and executed on NVIDIA H200 GPUs with CUDA 12.4, handling inputs typically around 8,000 tokens with a maximum context length of 10,240 tokens.
๐ Competitor Analysisโธ Show
| Benchmark | Task Type / Focus | Data Source | Scale / Items | Evaluation Focus |
|---|---|---|---|---|
| EHRBench | Clinical Decision Making (Diagnosis, Treatment, Prognosis) | Real-world EHR trajectories | Nearly 1M QA items | LLM performance in clinical inference, reliability, hallucination reduction |
| EHR-Bench (from EHR-R1) | Reasoning and Prediction across EHR scenarios (42 tasks) | MIMIC-IV, Stanford Medicine, Beth Israel Deaconess Medical Center | 42 tasks | Comprehensive assessment of reasoning and prediction capabilities |
| EHRStruct | Structured EHR tasks (11 tasks across 6 categories) | Synthea, eICU | 2,200 task-specific evaluation samples | LLM performance on structured EHR data, input formats, few-shot generalization |
| MedAgentBench | LLM Agent Capabilities in Medical Records (interactive) | De-identified real patient cases (STARR project) | 300 physician-written tasks, 100 patient profiles, 700k+ data points | Decision-making, planning, execution in FHIR-compliant virtual EHR environment |
| PhysicianBench | LLM Agents in Real EHR Environments (long-horizon tasks) | Real clinical consultation cases | 100 tasks (21 specialties), 670 checkpoints | Multi-step agent behavior, data retrieval, reasoning across encounters, clinical actions, documentation |
๐ ๏ธ Technical Deep Dive
- Construction Pipeline: EHRBench preprocesses and normalizes encounter-level clinical events into a standardized representation. It then generates structured templates by integrating EHR signals, LLM-based extraction, and knowledge-base (KB) verification and enrichment. A QA generation module deterministically instantiates each template into multiple QA items.
- Scalability and Reliability: The pipeline is LLM-driven for efficiency and scalability, KB-verified for reliability (filtering hallucinated or ambiguous relations), and EHR-grounded for clinical relevance.
- Specialized LLM Use: A specialized LLM is employed to automatically convert encounter-level EHR trajectories into structured templates.
- Verification Process: Systematic KB-based verification and enrichment are applied to filter out hallucinated or ambiguous relations and improve the reliability of the generated QA items.
- Evaluation Environment: Experiments are implemented in Python 3.11.5 and executed on NVIDIA H200 GPUs with CUDA 12.4.
- Input Handling: Questions are processed in batches of ten using instruction prompts. Inputs typically consist of around 8,000 tokens, with a maximum context length of 10,240 tokens.
- Efficiency: Early stopping is enabled once the model produces a complete JSON-formatted answer to reduce latency.
๐ฎ Future ImplicationsAI analysis grounded in cited sources
EHRBench will accelerate the development of more reliable and clinically relevant LLM systems for healthcare.
By providing a large-scale, automated, and reliable benchmark, it allows for consistent evaluation and identification of gaps in current LLM capabilities, guiding future research and development in medical AI.
The EHR-LLM-KB interaction pipeline methodology could be adapted for automated benchmark creation in other complex, data-rich domains beyond healthcare.
Its novel approach to generating scalable and reliable QA items from real-world data, verified by a knowledge base, offers a transferable framework for developing robust evaluation tools in various fields.
โณ Timeline
1960s
Early electronic health record (EHR) systems begin to emerge, with institutions like the Mayo Clinic pioneering their use.
2004
The Office of the National Coordinator (ONC) of Health Information Technology (IT) is created in the US to promote widespread EHR adoption.
2023-07
EHRSHOT, a benchmark for few-shot evaluation of foundation models on EHR data, is published, marking an early focus on LLMs with real-world clinical data.
2025-10
The EHR-R1 paper introduces EHR-Bench, a new benchmark curated from MIMIC-IV, designed to assess reasoning and prediction across 42 diverse EHR scenarios.
2026-04
EHRStruct, a comprehensive benchmark framework for evaluating LLMs on structured EHR tasks, is published, defining 11 tasks across 6 categories.
2026-05-28
EHRBench, a large-scale benchmark for clinical decision-making using real-world EHRs and an automated EHR-LLM-KB interaction pipeline, is published on arXiv.
๐ Sources (10)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
๐ฐ
Weekly AI Recap
Read this week's curated digest of top AI events โ
๐Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: ArXiv AI โ