Article

A statistical framework for evaluating the repeatability and reproducibility of large language models

2025-08-08

Abstract excerpt

<h4>Objective</h4> Systematic evaluation of variability in large language model (LLM)-generated outputs is critical for assessing their reliability. We developed a regulatory-informed statistical framework to quantify LLM repeatability and reproducibility. <h4>Materials and Methods</h4> Using definitions from U.S. Food and Drug Administration draft guidance for AI-enabled medical software, our framework quantifi...

Topics

Open a Topic to create a Post that cites this publication.

Identifiers and source

Literature Corpus work: b71575f4-e7b7-5c5e-a73e-f7da5757e094
DOI: 10.1101/2025.08.06.25333170

Open publication

Related research

Semantic proximity does not establish scientific evidence.

Click a neighbor to travelStep 1 · 12 closest

A statistical framework for evaluating the repeatability and reproducibility of large language modelsDOI 10.1101/2025.08.06.25333170Read

A Systematic Review of Testing and Evaluation of Healthcare Applications of Large Language Models (LLMs)Semantic proximity Read

High Consistency, Limited Accuracy: Evaluating Large Language Models for Binary Medical DiagnosisSemantic proximity Read

A Systematic Exploration of LLM Behavior for EHR PhenotypingSemantic proximity Read

Multidimensional Evaluation of Large Language Models on the AAP In-Service Examination: Assessing Accuracy, Calibration, and Citation ReliabilitySemantic proximity Read

Confidence Is Not Probability: Verification-Based Confidence Prediction for LLM-Generated Clinical SummariesSemantic proximity Read

Evaluating Large Language Models’ Performance in FDA Regulatory ScienceSemantic proximity Read

From Protocol to Analysis Plan: Development and Validation of a Large Language Model Pipeline for Statistical Analysis Plan Generation using Artificial Intelligence (SAPAI)Semantic proximity Read

Automated Evaluation of Large Language Model Response Concordance with Human Specialist Responses on Physician-to-Physician eConsult CasesSemantic proximity Read

Systematic Literature Review of LLM-Large Language Model in Medical: Digital Health, Technology and ApplicationsSemantic proximity Read

End-to-End Reliability of Automated Systems for Diagnostic Data Extraction: A Benchmark Study in Uro-Oncologic Evidence SynthesisSemantic proximity Read

Testing and Evaluation of Generative Large Language Models in Electronic Health Record Applications: A Systematic ReviewSemantic proximity Read

CARDBiomedBench: A Benchmark for Evaluating Large Language Model Performance in Biomedical ResearchSemantic proximity Read