Article

Risks of Using Large Language Models in Grading: LLMs and Humans Prefer LLM-Generated Writing Over Human’s but LLMs Show a Stronger Systematic Bias

2026-08-19

Abstract excerpt

<p>Recent studies suggest that large language models (LLMs) can approximate human grading performance, raising the prospect of their use in educational assessment. However, an important question remains: can LLMs evaluate student work fairly? In Study 1, we examined whether LLM evaluators exhibit systematic preference when grading psychology dissertations from three sources: (a) dissertations authored by 1,426 und...

Topics

Open a Topic to create a Post that cites this publication.

Identifiers and source

Literature Corpus work: dd6418d8-9c4f-5f31-801a-1ead0a4b87f0
DOI: 10.31234/osf.io/35utw_v1

Open publication

Related research

Semantic proximity does not establish scientific evidence.

Click a neighbor to travelStep 1 · 12 closest

Risks of Using Large Language Models in Grading: LLMs and Humans Prefer LLM-Generated Writing Over Human’s but LLMs Show a Stronger Systematic BiasDOI 10.31234/osf.io/35utw_v1Read

Integrating large language models into the psychological sciencesSemantic proximity Read

Looks good on paper: LLM-generated exams are face-valid but psychometrically weaker than human assessmentsSemantic proximity Read

Using GPT-4 to Automate the Generation of Lay Summaries for Cancer PublicationsSemantic proximity Read

Natural Language Processing and Generative AI in the Automated Scoring and Feedback of Reflective Writing in Medical Education: A Validity and Fairness AnalysisSemantic proximity Read

Can ChatGPT Replace the Teacher in Assessment? A Review of Research on the Use of Large Language Models in Grading and Providing FeedbackSemantic proximity Read

Can GPT-4 Replace Human Examiners? A Competition on Checking Open-Text AnswersSemantic proximity Read

Can Large Language Models Emulate Human Performance on Educational Assessments?Semantic proximity Read

Large Language Model–Generated Tutoring Responses Receive Higher Ratings but Are Penalized by Source-Based Evaluation BiasSemantic proximity Read

Using Large Language Models to Augment (Rather Than Replace) Human Feedback in Higher Education Improves Perceived Feedback QualitySemantic proximity Read

Can AI Match Human Experts? Evaluating LLM-Generated Feedback on Resident Scholarly ProjectsSemantic proximity Read