Article

CalibJudge: Calibrated LLM-as-a-Judge for Multilingual RAG with Uncertainty-Aware Scoring

2026-03-17

Abstract excerpt

Large Language Models (LLMs) serving as automatic evaluators (LLM-as-a-Judge) have become essential for assessing Retrieval-Augmented Generation (RAG) systems. However, in multilingual settings, these judges exhibit significant calibration drift across languages, producing scores that are neither comparable nor aligned with human judgments. We present CalibJudge, a post-hoc calibration framework that addresses thi...

Topics

Open a Topic to create a Post that cites this publication.

Uncategorized literature

Identifiers and source

Literature Corpus work: 51791edd-5439-5e89-b620-ce33b982c26d
DOI: 10.20944/preprints202603.1324.v1

Open publication

Related research

Semantic proximity does not establish scientific evidence.

Click a neighbor to travelStep 1 · 12 closest

CalibJudge: Calibrated LLM-as-a-Judge for Multilingual RAG with Uncertainty-Aware ScoringDOI 10.20944/preprints202603.1324.v1Read

Adaptive Rigor in AI System Evaluation using Temperature-Controlled Verdict Aggregation via Generalized Power MeanSemantic proximity Read

Adaptive Rigor in AI System Evaluation using Temperature-Controlled Verdict Aggregation via Generalized Power MeanSemantic proximity Read

RePCS: Diagnosing Data Memorization in LLM-Powered Retrieval-Augmented GenerationSemantic proximity Read

A Survey of Large Language Models: Evolution, Architectures, Adaptation, Benchmarking, Applications, Challenges, and Societal ImplicationsSemantic proximity Read

Optimizing Human-Controlled Preference Alignment in Large Language Models via Dense Token Masking: A Methodological ApproachSemantic proximity Read

Fine-grained Debiasing for Large Language Modelsvia Bias Intensity and Probability DecouplingSemantic proximity Read

DAFE: LLM-Based Evaluation Through Dynamic Arbitration for Free-Form Question-AnsweringSemantic proximity Read

MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation SystemsSemantic proximity Read

A Multi-Model, Multi-Domain Benchmark of Large Language Model Agreement with Humans and with Each Other in Sentiment ClassificationSemantic proximity Read

Adapting Large Language Models for Low-Resource Regulated Domains: A Fine-Tuning and Retrieval-Augmented Generation Approach to Insurance Information Delivery in KenyaSemantic proximity Read

Uncertainty-Aware Assessment of LLM-Enhanced Topic Models: An Experton-Based Approach to InterpretabilitySemantic proximity Read

Improved Value Alignment in Large Language Models Using Variational Best-of-N TechniquesSemantic proximity Read