Article

Aggregate benchmark scores obscure patient safety implications of errors across frontier language models

2026-03-20

Abstract excerpt

Frontier language models are widely used for health-related queries, yet aggregate benchmark scores do not capture safety implications of errors. We applied the recent Nature Medicine triage benchmark across nine frontier models, comparing directional error profiles, contextual bias, and crisis calibration. In-range accuracy ranged from 75.0% to 87.7%, obscuring clinically meaningful error differences. Looking at...

Topics

Open a Topic to create a Post that cites this publication.

Identifiers and source

Literature Corpus work: 45784e96-f1dd-5727-b68e-af461ad8a57b
DOI: 10.64898/2026.03.18.26348695

Open publication

Related research

Semantic proximity does not establish scientific evidence.

Click a neighbor to travelStep 1 · 12 closest

Aggregate benchmark scores obscure patient safety implications of errors across frontier language modelsDOI 10.64898/2026.03.18.26348695Read

Medical errors in large language models revealed using 1,000 synthetic clinical transcriptsSemantic proximity Read

Performance evaluation and benchmarking across 16 large language models on a comprehensive real-world emergency department triage data setSemantic proximity Read

Beyond Benchmarks: Dynamic, Automatic And Systematic Red-Teaming Agents For Trustworthy Medical Language ModelsSemantic proximity Read

Evaluating Sycophancy in Frontier Models Using Persona-Driven ChallengeSemantic proximity Read

Performance evaluation and benchmarking across 16 large language models on a comprehensive real-world emergency department triage data setSemantic proximity Read

Cross-Model Variability in Large Language Model Triage Behavior for Potential Stroke SymptomsSemantic proximity Read

AI safety evaluation in an underrepresented population: real-world performance of clinical decision support and frontier language models on Medicaid patient messaging triageSemantic proximity Read

An interdisciplinary, randomized, single-blind evaluation of state-of-the-art large language models for their implications and risks in medical diagnosis and managementSemantic proximity Read

MTS-Bench: A Manchester Triage System Benchmark for Language Model Triage SafetySemantic proximity Read

Dissecting clinical reasoning failures in frontier artificial intelligence using 10,000 synthetic casesSemantic proximity Read

Vertical Model Paradox: Systemic Ethical Blind Spots in Domain-Specific Medical AISemantic proximity Read

Impact of LLM Assistance on Physician Decision-Making: A Multi-Country Randomized Controlled Trial <sup>∗</sup>Semantic proximity Read