Article

Validating local open-weight LLMs against expert raters for graded deductive coding of open-ended responses in Russian

2026-08-07

Abstract excerpt

<p>Large language models (LLMs) are increasingly used to code open-ended survey responses, but their validity against expert annotation remains uncertain, especially outside English. We evaluated four open-weight LLMs for deductive coding of Russian responses from a psychological human-robot interaction experiment (N = 33): DeepSeek-R1-32B, Qwen3-32B, Gemma3-27B, and Mistral-Small-3.2-24B. Ten psychology-trained c...

Topics

Open a Topic to create a Post that cites this publication.

Genomics and Rare Diseases

Identifiers and source

Literature Corpus work: 22db7a3b-392c-5950-8c47-7378abc2e96e
DOI: 10.31234/osf.io/8psd9_v2

Open publication

Related research

Semantic proximity does not establish scientific evidence.

Click a neighbor to travelStep 1 · 12 closest

Validating local open-weight LLMs against expert raters for graded deductive coding of open-ended responses in RussianDOI 10.31234/osf.io/8psd9_v2Read

chatRater: Validating LLM-Generated Psycholinguistic Norms with Probability-Weighted ScoringSemantic proximity Read

Are LLMs better expert judges? Rethinking content validity assessment in the age of AISemantic proximity Read

Large Language Models in Psychology: Application in the Context of a Systematic Literature Review.Semantic proximity Read

When ChatGPT-4o Is (Less) Human-Like: Preliminary Subjective Rating Tests for Psycholinguistic ResearchSemantic proximity Read

Evaluating the strengths and weaknesses of large language models in answering neurophysiology questionsSemantic proximity Read

Artificial intelligence in psychology researchSemantic proximity Read

Can AI Be Morally Wise? Evidence of a Systematic Alignment Artifact from a Validated Human BaselineSemantic proximity Read

Private Local Large Language Model for Generating Context-Aware Responses About the Regulations of Book Four of the Civil Code of Catalonia and the Notary Profession of SpainSemantic proximity Read

Nonlinear Transformation of Probabilities by Large Language ModelsSemantic proximity Read

Model shopping: how answer selection across AI models inflates scientific error ratesSemantic proximity Read

The Moral Turing Test: Evaluating Human-LLM Alignment in Moral Decision-MakingSemantic proximity Read