Article

Looks good on paper: LLM-generated exams are face-valid but psychometrically weaker than human assessments

2026-03-26

Abstract excerpt

<p>Practice exams support learning but are labor-intensive to create, prepare, and implement. Large language models (LLMs) may help to reduce this burden, but the quality of LLM-generated assessment items has varied in research to date. Across two studies, we evaluate PREPARE, a comprehensive LLM-based workflow aimed at high-quality multiple-choice question generation. In Study 1, we examined the perceived relativ...

Topics

Open a Topic to create a Post that cites this publication.

Uncategorized literature

Identifiers and source

Literature Corpus work: 65d90ed7-6801-5ded-a91c-8bf42cc98fa2
DOI: 10.31234/osf.io/vaf2y_v1

Open publication

Related research

Semantic proximity does not establish scientific evidence.

Click a neighbor to travelStep 1 · 12 closest

Looks good on paper: LLM-generated exams are face-valid but psychometrically weaker than human assessmentsDOI 10.31234/osf.io/vaf2y_v1Read

Psychometric Performance and Student Perceptions of AI- versus Human-Generated Multiple-Choice Questions: The AHEAD Randomized Controlled TrialSemantic proximity Read

Expert validation of AI-generated clinical assessment items for physician assistant certification: a multi-dimensional rubric and dual- reliability methodSemantic proximity Read

Large language models for generating medical examinations: systematic reviewSemantic proximity Read

Generating Better Items for Cognitive Assessments Using Large Language ModelsSemantic proximity Read

Field-testing multiple-choice questions with AI examinees: English grammar itemsSemantic proximity Read

Large language models for generating medical examinations: systematic reviewSemantic proximity Read

Quality assurance and validity of AI-generated Single Best Answer questionsSemantic proximity Read

GPT-4 versus human authors in clinically complex MCQ creation: a blinded analysis of item qualitySemantic proximity Read

Risks of Using Large Language Models in Grading: LLMs and Humans Prefer LLM-Generated Writing Over Human’s but LLMs Show a Stronger Systematic BiasSemantic proximity Read

QUEST-AI: A System for Question Generation, Verification, and Refinement using AI for USMLE-Style ExamsSemantic proximity Read

Can Large Language Models Emulate Human Performance on Educational Assessments?Semantic proximity Read

Can Large Language Models Emulate Human Performance on Educational Assessments?Semantic proximity Read