Article

Across Generations, Sizes, and Types, Large Language Models Poorly Report Self-Confidence in Gastroenterology Clinical Reasoning Tasks

2025-06-04

Abstract excerpt

<title>Abstract</title> <p>This study evaluated confidence calibration across 48 large language models (LLM) using 300 gastroenterology board exam style questions. Regardless of response accuracy, all models demonstrated poor certainty estimation. Even the best-calibrated systems (o1 preview, GPT-4o, Claude-3.5-Sonnet) showed substantial overconfidence (Brier scores 0.15-0.2, AUROC ~0.6). Most concerning, models...

Topics

Open a Topic to create a Post that cites this publication.

Identifiers and source

Literature Corpus work: 16dffec3-5bf2-5b66-aba2-c1b0919d3c14
DOI: 10.21203/rs.3.rs-6725427/v1

Open publication

Related research

Semantic proximity does not establish scientific evidence.

Click a neighbor to travelStep 1 · 12 closest

Across Generations, Sizes, and Types, Large Language Models Poorly Report Self-Confidence in Gastroenterology Clinical Reasoning TasksDOI 10.21203/rs.3.rs-6725427/v1Read

Benchmarking the Confidence of Large Language Models in Clinical QuestionsSemantic proximity Read

General purpose large language models match human performance on gastroenterology board exam self-assessmentsSemantic proximity Read

Comparative Performance of General-Purpose Large Language Models and a Specialized Clinical AI Tool on OKAP-Style Ophthalmology QuestionsSemantic proximity Read

Multimodal Large Language Model Passes Specialty Board Examination and Surpasses Human Test-Taker Scores: A Comparative Analysis Examining the Stepwise Impact of Model Prompting Strategies on PerformanceSemantic proximity Read

Textbook-Level Medical Knowledge in Large Language Models: A Comparative Evaluation Using the Japanese National Medical ExaminationSemantic proximity Read

Performance of Advanced Large Language Models (GPT-4o, GPT-4, Gemini 1.5 Pro, Claude 3 Opus) on Japanese Medical Licensing Examination: A Comparative StudySemantic proximity Read

Comparison of Large Language Models’ Performance on Neurosurgical Board Examination QuestionsSemantic proximity Read

Board-Level Performance of Leading Open-Weight Vision-Language Models on the Japanese Diagnostic Radiology Board Examination: Reasoning, Image-Input, and Language EffectsSemantic proximity Read

Performance of o1 pro and GPT-4 in self-assessment questions for nephrology board renewalSemantic proximity Read

High Consistency, Limited Accuracy: Evaluating Large Language Models for Binary Medical DiagnosisSemantic proximity Read

Comparative Evaluation the Knowledge of Large Language Models about Response Evaluation Criteria in Solid Tumors?Semantic proximity Read

Evaluation of Large Language Models in the Clinical Management of Patients With Upper Gastrointestinal Bleeding : Insights From Real-World Patient DataSemantic proximity Read