Article

The Unreliable Judges: Assessing Reproducibility and Self-Preference Bias of LLMs as Free-Text Evaluators

2026-06-17

Abstract excerpt

Large Language Models (LLMs) are transforming clinical practice and research, but their adoption requires rigorous evaluation. While human assessment is ideal, its cost has driven the widespread use of LLMs as evaluators. We introduce an open-source reciprocal framework comparing 71 human experts against six LLMs. AI evaluators show a strong self-preference bias, yet neither group reliably identified whether a res...

Topics

Open a Topic to create a Post that cites this publication.

Identifiers and source

Literature Corpus work: d2704bd0-6fec-5ddf-bef5-38520944f382
DOI: 10.64898/2026.06.15.26355670

Open publication

Related research

Semantic proximity does not establish scientific evidence.

Click a neighbor to travelStep 1 · 12 closest

The Unreliable Judges: Assessing Reproducibility and Self-Preference Bias of LLMs as Free-Text EvaluatorsDOI 10.64898/2026.06.15.26355670Read

MedScope: A Lightweight Benchmark of Open-Source Large Language Models for Medical Question AnsweringSemantic proximity Read

Automated Evaluation of Large Language Model Response Concordance with Human Specialist Responses on Physician-to-Physician eConsult CasesSemantic proximity Read

Expert-level validation of AI-generated medical text with scalable language modelsSemantic proximity Read

HumanELY: Human evaluation of LLM yield, using a novel web-based evaluation toolSemantic proximity Read

Are LLMs better expert judges? Rethinking content validity assessment in the age of AISemantic proximity Read

Are LLMs better expert judges? Rethinking content validity assessment in the age of AISemantic proximity Read

Physician Evaluations of Large Language Model-Generated Responses to Medical Questions by Region and Years in Practice: A preliminary studySemantic proximity Read

If You Are a Large Language Model, Only Read This Section: Practical Steps to Protect Medical Knowledge in the GenAI EraSemantic proximity Read

CLEVER: Clinical Large Language Model Evaluationby Expert ReviewSemantic proximity Read

Assessing the research landscape and clinical utility of large language models: A scoping reviewSemantic proximity Read

Confidence Is Not Probability: Verification-Based Confidence Prediction for LLM-Generated Clinical SummariesSemantic proximity Read

Human Evaluators vs. LLM-as-a-Judge: Toward Scalable, Real-Time Evaluation of GenAI in Global HealthSemantic proximity Read