Article

Human Evaluators vs. LLM-as-a-Judge: Toward Scalable, Real-Time Evaluation of GenAI in Global Health

2025-10-28

Abstract excerpt

Evaluating the outputs of generative AI (GenAI) models in healthcare remains a significant bottleneck for the safe and scalable deployment of these tools. Human expert raters remain the gold standard for assessing the accuracy, contextual appropriateness, and empathy of AI-generated responses, but their assessments are costly, inconsistent, and difficult to scale. The concept of “LLM-as-a-judge” systems, i.e., AI...

Topics

Open a Topic to create a Post that cites this publication.

Identifiers and source

Literature Corpus work: 8e132e20-9b47-53f9-938e-371ed6ccdf75
DOI: 10.1101/2025.10.27.25338910

Open publication

Related research

Semantic proximity does not establish scientific evidence.

Click a neighbor to travelStep 1 · 12 closest

Human Evaluators vs. LLM-as-a-Judge: Toward Scalable, Real-Time Evaluation of GenAI in Global HealthDOI 10.1101/2025.10.27.25338910Read

Impact of LLM Assistance on Physician Decision-Making: A Multi-Country Randomized Controlled Trial <sup>∗</sup>Semantic proximity Read

Reproducible Generative AI Evaluation for Healthcare: A Clinician-in-the-Loop ApproachSemantic proximity Read

Bridging AI and Healthcare: A Scoping Review of Retrieval-Augmented Generation—Ethics, Bias, Transparency, Improvements, and ApplicationsSemantic proximity Read

Automated Evaluation of Large Language Model Response Concordance with Human Specialist Responses on Physician-to-Physician eConsult CasesSemantic proximity Read

Benchmarking Large Language Models and Clinicians Using Locally Generated Primary Healthcare Vignettes in KenyaSemantic proximity Read

Performance of Large Language Models as a Tool for Primary Care Consultations: Evaluation StudySemantic proximity Read

Physician Evaluations of Large Language Model-Generated Responses to Medical Questions by Region and Years in Practice: A preliminary studySemantic proximity Read

Natural Language Processing and Generative AI in the Automated Scoring and Feedback of Reflective Writing in Medical Education: A Validity and Fairness AnalysisSemantic proximity Read

A Review of a Hybrid Evaluation Framework for Clinical AI: Integrating Interrater Reliability with the "LLM as a Judge" MethodologySemantic proximity Read

Vertical Model Paradox: Systemic Ethical Blind Spots in Domain-Specific Medical AISemantic proximity Read

ACCORD-AI: an exploratory evaluation of large language models for drafting consensus voting itemsSemantic proximity Read

Evaluating Sycophancy in Frontier Models Using Persona-Driven ChallengeSemantic proximity Read