Back to search

Article

Beyond Mean Scores: Individual- and Trait-Level Agreement Between Human and Generative AI Raters in EFL Writing Assessment

2026-08-28

Abstract excerpt

<title>Abstract</title> <p>Large language models (LLMs) are increasingly proposed as writing assessors, yet comparisons based only on group means or correlations can conceal disagreement for individual scripts and analytic traits. This study compared human raters with Claude Fable 5, GPT 5.6, and Gemini 3.1 Flash in an end-to-end, scan-based evaluation of a consequential EFL proficiency writing assessment. Sevent...

Topics

Open a Topic to create a Post that cites this publication.

Identifiers and source

Literature Corpus work
e9c4b2df-3254-5d7c-9015-c9af2dc7abc0
DOI
10.21203/rs.3.rs-10728298/v1
Open publication

Related research

Semantic proximity does not establish scientific evidence. Graph snapshot: 2026-09-29.

Click a neighbor to travelStep 1 · 12 closest
Interactive article relationship graphSelect a related publication card to move it into the centre and load its closest explainable connections. Solid lines are source-backed structured connections. Dashed lines are semantic discovery signals and are not scientific evidence.
Beyond Mean Scores: Individual- and Trait-Level Agreement Between Human and Generative AI Raters in EFL Writing AssessmentDOI 10.21203/rs.3.rs-10728298/v1
Beyond Automated Scoring: An Evidence-Informed Human–GenAI Complementarity Framework for Formative Writing AssessmentSemantic proximity
The AI Paradox in L2 Writing: Why Helpful Feedback Creates Unhelpful Dependency in Higher EducationSemantic proximity
Natural Language Processing and Generative AI in the Automated Scoring and Feedback of Reflective Writing in Medical Education: A Validity and Fairness AnalysisSemantic proximity
The AI Paradox in L2 Writing: Why Helpful Feedback Creates Unhelpful Dependency in Higher EducationSemantic proximity
Calibrating Gemini to Human Raters in Spoken Language Assessment for EFL Learners in AsiaSemantic proximity
Can AI replace Teachers? Comparing AI and Human Teachers’ ratings and feedback on students’ essaysSemantic proximity
Risks of Using Large Language Models in Grading: LLMs and Humans Prefer LLM-Generated Writing Over Human’s but LLMs Show a Stronger Systematic BiasSemantic proximity
A Review of a Hybrid Evaluation Framework for Clinical AI: Integrating Interrater Reliability with the "LLM as a Judge" MethodologySemantic proximity
A Human-AI Integrated Rating System for Improving Second Language Writing:The Case of Japanese Learners of English for General Academic PurposesSemantic proximity
Large Language Model–Generated Tutoring Responses Receive Higher Ratings but Are Penalized by Source-Based Evaluation BiasSemantic proximity
Large Language Model–Generated Tutoring Responses Receive Higher Ratings but Are Penalized by Source-Based Evaluation BiasSemantic proximity
AI-Generated Clinical Summaries: Errors and Susceptibility to Speech and Speaker VariabilitySemantic proximity
Select a neighboring publication to make it the new centre.