Article

Context Matching is not Reasoning: Assessing Generalized Evaluation of Generative Language Models in Clinical Settings

2025-08-29

Abstract excerpt

<title>Abstract</title> <p>Current discussion surrounding the clinical capabilities of generative language models (GLMs) predominantly center around multiple-choice question-answer (MCQA) benchmarks derived from clinical licensing examinations. While accepted for human examinees, characteristics unique to GLMs bring into question the validity of such benchmarks. Here, we validate four benchmarks using eight GLMs,...

Topics

Open a Topic to create a Post that cites this publication.

Identifiers and source

Literature Corpus work: 9199db4c-9fa8-5bb4-ad55-7574c6465053
DOI: 10.21203/rs.3.rs-7325383/v1

Open publication

Related research

Semantic proximity does not establish scientific evidence.

Click a neighbor to travelStep 1 · 12 closest

Context Matching is not Reasoning: Assessing Generalized Evaluation of Generative Language Models in Clinical SettingsDOI 10.21203/rs.3.rs-7325383/v1Read

Understanding the Inner Workings of Large Language Models in MedicineSemantic proximity Read

Quality assurance and validity of AI-generated Single Best Answer questionsSemantic proximity Read

Characterizing large language model generative artificial intelligence variability in the production of objective structured clinical examination stationsSemantic proximity Read

Multidimensional Evaluation of Large Language Models on the AAP In-Service Examination: Assessing Accuracy, Calibration, and Citation ReliabilitySemantic proximity Read

The Economics of Accuracy for Medical Reasoning with Large Language ModelsSemantic proximity Read

MISP-Bench: Decomposing User-Provided False Priors into Answer, Rationale, and Guard EffectsSemantic proximity Read

Field-testing multiple-choice questions with AI examinees: English grammar itemsSemantic proximity Read

Retrieval-Augmented-Generation large language models outperform junior clinicians in guideline-concordant PSA testingSemantic proximity Read

Chat-WanJuan: A Fine-Tuned Compact Language Model for Multiple- Choice Question Generation in Medical EducationSemantic proximity Read

Multimodal Large Language Model Passes Specialty Board Examination and Surpasses Human Test-Taker Scores: A Comparative Analysis Examining the Stepwise Impact of Model Prompting Strategies on PerformanceSemantic proximity Read

AI-Powered Test Question Generation in Medical Education: The DailyMed ApproachSemantic proximity Read

Are LLMs better expert judges? Rethinking content validity assessment in the age of AISemantic proximity Read