Article

chatRater: Validating LLM-Generated Psycholinguistic Norms with Probability-Weighted Scoring

2026-05-16

Abstract excerpt

<p>Large language models (LLMs) offer a scalable alternative to labour-intensive human normingstudies in psycholinguistic research. This article introduces chatRater, an R package for ratingtext, image, and audio stimuli via multiple LLM providers, and demonstrates its use with a casestudy validating GPT-4o ratings of English idioms against human norms. We extend theprobability-weighted scoring method of Brysbaert...

Topics

Open a Topic to create a Post that cites this publication.

Uncategorized literature

Identifiers and source

Literature Corpus work: c5954f5a-ec6c-52ef-a1e9-a879fd1677e2
DOI: 10.31234/osf.io/mje6w_v1

Open publication

Related research

Semantic proximity does not establish scientific evidence.

Click a neighbor to travelStep 1 · 12 closest

chatRater: Validating LLM-Generated Psycholinguistic Norms with Probability-Weighted ScoringDOI 10.31234/osf.io/mje6w_v1Read

When ChatGPT-4o Is (Less) Human-Like: Preliminary Subjective Rating Tests for Psycholinguistic ResearchSemantic proximity Read

Can Large Language Models Help Augment English Psycholinguistic Datasets?Semantic proximity Read

Validating LLMs as Coders of Cultural Constructs: a Large-Scale Test of Minimally Counterintuitiveness using PokémonSemantic proximity Read

Validating local open-weight LLMs against expert raters for graded deductive coding of open-ended responses in RussianSemantic proximity Read

Perils and Opportunities in Using Large Language Models in Psychological ResearchSemantic proximity Read

Contextual Assembly of Lexical Functions in Large Language ModelsSemantic proximity Read

Emergent Mental Lexicon Functions in ChatGPTSemantic proximity Read

Can AI Be Morally Wise? Evidence of a Systematic Alignment Artifact from a Validated Human BaselineSemantic proximity Read

A Comparative Study of Cultural Hallucination in Large Language Models on Culturally Specific Ethical QuestionsSemantic proximity Read

Large Language Models and the Wisdom of Small CrowdsSemantic proximity Read

Are LLMs better expert judges? Rethinking content validity assessment in the age of AISemantic proximity Read

Are LLMs better expert judges? Rethinking content validity assessment in the age of AISemantic proximity Read