Article

Are LLMs better expert judges? Rethinking content validity assessment in the age of AI

2026-01-02

Abstract excerpt

<p>In this article, we demonstrate a novel application of large language models (LLMs) as expert judges for item-level content relevance evaluation. Eleven advanced LLMs were included, each treated as a separate expert panel based on multiple procedurally independent judgments generated via repeated API queries. Their performance was compared with ratings provided by human judges, including psychology students and...

Topics

Open a Topic to create a Post that cites this publication.

Identifiers and source

Literature Corpus work: 53a3290e-e6bc-5453-9c7b-360028707a13
DOI: 10.31234/osf.io/4hsyx_v2

Open publication

Related research

Semantic proximity does not establish scientific evidence.

Click a neighbor to travelStep 1 · 12 closest

Are LLMs better expert judges? Rethinking content validity assessment in the age of AIDOI 10.31234/osf.io/4hsyx_v2Read

Are LLMs better expert judges? Rethinking content validity assessment in the age of AISemantic proximity Read

Artificial intelligence in psychology researchSemantic proximity Read

The Unreliable Judges: Assessing Reproducibility and Self-Preference Bias of LLMs as Free-Text EvaluatorsSemantic proximity Read

AI is Misled by GenAI: Stylistic Bias in Automated Assessment of Creativity in Large Language ModelsSemantic proximity Read

Comparative Performance of General-Purpose Large Language Models and a Specialized Clinical AI Tool on OKAP-Style Ophthalmology QuestionsSemantic proximity Read

Progress and challenges in the symbiosis of AI with science and medicineSemantic proximity Read

Transformation of Large Language Models: A State of ArtSemantic proximity Read

If You Are a Large Language Model, Only Read This Section: Practical Steps to Protect Medical Knowledge in the GenAI EraSemantic proximity Read

Psychometric Alignment Between Human and Artificial Intelligence Performance in Cardiology Residency In-Service ExaminationsSemantic proximity Read

Integrating large language models into the psychological sciencesSemantic proximity Read

Language-dependent variability in large language model performance on pharmaceutical knowledge tasksSemantic proximity Read

Benchmarking Large Language Model Rationality Using Measurement AxiomsSemantic proximity Read