Article

RheumBench: A Specialist-Validated Rubric-Based Benchmark for Evaluating Large Language Models in Rheumatology

2026-09-01

Abstract excerpt

<title>Abstract</title> <p>Large language models (LLMs) are increasingly used for clinical decision support, yet scalable and specialty-specific methods for evaluating their performance and safety remain limited. We developed RheumBench, the first rubric-based LLM benchmark in rheumatology, providing a specialist-validated framework for open-ended evaluation. We evaluated ten systems, comprising three frontier pr...

Identifiers and source

Literature Corpus work: 465b7d4c-767f-5416-a9e3-79b858bc5b15
DOI: 10.21203/rs.3.rs-10825548/v1

Open publication

Related research

Semantic proximity does not establish scientific evidence. Graph snapshot: 2026-09-29.

Click a neighbor to travelStep 1 · 12 closest

RheumBench: A Specialist-Validated Rubric-Based Benchmark for Evaluating Large Language Models in RheumatologyDOI 10.21203/rs.3.rs-10825548/v1Read

Diagnostic Accuracy of Large Language Models for Rare Diseases: A Systematic Review and Meta-AnalysisSemantic proximity Read

Advancing Rheumatology Practice: Systematic Review of Natural Language Processing ApplicationsSemantic proximity Read

A systematic review and meta-analysis of the Diagnostic Performance of Large Language Models in Rare DiseasesSemantic proximity Read

High Consistency, Limited Accuracy: Evaluating Large Language Models for Binary Medical DiagnosisSemantic proximity Read

Optimizing the Clinical Application of Rheumatology Guidelines Using Large Language Models: A Retrieval-Augmented Generation Framework Integrating EULAR and ACR RecommendationsSemantic proximity Read

Assessing and optimizing large language models on spondyloarthritis multi-choice question answering (SpAMCQA): study protocol for a bilingual evaluation benchmarkSemantic proximity Read

Evaluating large language model workflows in clinical decision support: referral, triage, and diagnosisSemantic proximity Read

A Systematic Review of Large Language Models in Medical Specialties: Applications, Challenges and Future DirectionsSemantic proximity Read

A Systematic Review of Testing and Evaluation of Healthcare Applications of Large Language Models (LLMs)Semantic proximity Read

Benchmarking the Performance of Large Language Models in Uveitis: A Comparative Analysis of ChatGPT-3.5, ChatGPT-4.0, Google Gemini, and Anthropic Claude3Semantic proximity Read

The TRIPOD-LLM Statement: A Targeted Guideline For Reporting Large Language Models UseSemantic proximity Read

Automated Evaluation of Large Language Model Response Concordance with Human Specialist Responses on Physician-to-Physician eConsult CasesSemantic proximity Read