Article

Benchmarking the Confidence of Large Language Models in Clinical Questions

2024-08-11

Abstract excerpt

<h4>Background and Aim</h4> The capabilities of large language models (LLMs) to self-assess their own confidence in answering questions in the biomedical realm remain underexplored. This study evaluates the confidence levels of 12 LLMs across five medical specialties to assess their ability to accurately judge their responses. <h4>Methods</h4> We used 1,965 multiple-choice questions assessing clinical knowledge...

Topics

Open a Topic to create a Post that cites this publication.

Identifiers and source

Literature Corpus work: b2cb4e00-ebcf-5753-b82e-8090e7123233
DOI: 10.1101/2024.08.11.24311810

Open publication

Related research

Semantic proximity does not establish scientific evidence.

Click a neighbor to travelStep 1 · 12 closest

Benchmarking the Confidence of Large Language Models in Clinical QuestionsDOI 10.1101/2024.08.11.24311810Read

Comparative Performance of General-Purpose Large Language Models and a Specialized Clinical AI Tool on OKAP-Style Ophthalmology QuestionsSemantic proximity Read

Across Generations, Sizes, and Types, Large Language Models Poorly Report Self-Confidence in Gastroenterology Clinical Reasoning TasksSemantic proximity Read

Physician Evaluations of Large Language Model-Generated Responses to Medical Questions by Region and Years in Practice: A preliminary studySemantic proximity Read

Performance of reasoning large language models on nephrology multiple-choice questionsSemantic proximity Read

Comparing Human and Large Language Model Responses to Patients’ Online Questions: Towards Multi-dimensional Patient-centered SupportSemantic proximity Read

Multidimensional Evaluation of Large Language Models on the AAP In-Service Examination: Assessing Accuracy, Calibration, and Citation ReliabilitySemantic proximity Read

Evaluating Capabilities of Large Language Models: Performance of GPT4 on Surgical Knowledge AssessmentsSemantic proximity Read

Performance of o1 pro and GPT-4 in self-assessment questions for nephrology board renewalSemantic proximity Read

Large Language Models for the assessment of medical students’ clinical decision-makingSemantic proximity Read

Exploring the Capabilities of ChatGPT in Women’s HealthSemantic proximity Read

ARE LLMS READY FOR PEDIATRICS? A COMPARATIVE EVALUATION OF MODEL ACCURACY ACROSS CLINICAL DOMAINSSemantic proximity Read

Comparison of Large Language Models’ Performance on Neurosurgical Board Examination QuestionsSemantic proximity Read