Back to search

Article

Rater choice determines measured fidelity: a multi-model evaluation of large language model raters for AI-generated plain-language biomedical summaries

2026-08-27

Abstract excerpt

<title>Abstract</title> <p>Large language models are increasingly used to produce plain-language summaries of biomedical research and are also being suggested as the raters to check these summaries. The purpose of my study was to determine whether different rater models, when using the same source-anchored rubric, agree in their assessments of the accuracy of identical summaries and how often summaries fail to in...

Topics

Open a Topic to create a Post that cites this publication.

Identifiers and source

Literature Corpus work
42bbb44e-14ac-5369-a9c0-74c89114d7cd
DOI
10.21203/rs.3.rs-10822636/v1
Open publication

Related research

Semantic proximity does not establish scientific evidence. Graph snapshot: 2026-09-29.

Click a neighbor to travelStep 1 · 12 closest
Interactive article relationship graphSelect a related publication card to move it into the centre and load its closest explainable connections. Solid lines are source-backed structured connections. Dashed lines are semantic discovery signals and are not scientific evidence.
Rater choice determines measured fidelity: a multi-model evaluation of large language model raters for AI-generated plain-language biomedical summariesDOI 10.21203/rs.3.rs-10822636/v1
Confidence Is Not Probability: Verification-Based Confidence Prediction for LLM-Generated Clinical SummariesSemantic proximity
The use of a large language models to create plain language summaries of evidence reviews in healthcare: a feasibility studySemantic proximity
If You Are a Large Language Model, Only Read This Section: Practical Steps to Protect Medical Knowledge in the GenAI EraSemantic proximity
Assessing the Limitations of Large Language Models in Clinical Practice Guideline-concordant Treatment Decision-making on Real-world DataSemantic proximity
Can ChatGPT write better scientific titles? A comparative evaluation of human-written and AI-generated titlesSemantic proximity
Physician- and Large Language Model-Generated Hospital Discharge Summaries: A Blinded, Comparative Quality and Safety StudySemantic proximity
Can ChatGPT write better scientific titles? A comparative evaluation of human-written and AI-generated titlesSemantic proximity
Physician Evaluations of Large Language Model-Generated Responses to Medical Questions by Region and Years in Practice: A preliminary studySemantic proximity
RAPID: Reliable and efficient Automatic generation of submission rePorting checklists with Large language moDelsSemantic proximity
Physician- versus Large Language Model-Generated Clinical Summaries in the Emergency DepartmentSemantic proximity
Clinical Safety of Large Language Models in Oral Cancer–Related Patient Communication: A Longitudinal StudySemantic proximity
Evaluating Large Language Models for Drafting Emergency Department Discharge SummariesSemantic proximity
Select a neighboring publication to make it the new centre.