Article

Benchmarking Large Language Models and Clinicians Using Locally Generated Primary Healthcare Vignettes in Kenya

2025-10-27

Abstract excerpt

<h4>Background</h4> Large language models (LLMs) show promise on healthcare tasks, yet most evaluations emphasize multiple-choice accuracy rather than open-ended reasoning. Evidence from low-resource settings remains limited. <h4>Methods</h4> We benchmarked five LLMs (GPT-4.1, Gemini-2.5-Flash, DeepSeek-R1, MedGemma, and o3) against Kenyan clinicians, using a randomly subsampled dataset of 507 vignettes (from a...

Topics

Open a Topic to create a Post that cites this publication.

Identifiers and source

Literature Corpus work: d5c38d83-032e-5b49-8dd8-86922ec47147
DOI: 10.1101/2025.10.25.25338798

Open publication

Related research

Semantic proximity does not establish scientific evidence.

Click a neighbor to travelStep 1 · 12 closest

Benchmarking Large Language Models and Clinicians Using Locally Generated Primary Healthcare Vignettes in KenyaDOI 10.1101/2025.10.25.25338798Read

Can Large Language Models Provide High-Quality Information for Patients with Multidrug-Resistant Organism Infections: A Multidimensional Comparative Analysis of DeepSeek-V3, ChatGPT-5.2, Gemini-3, and Grok-4Semantic proximity Read

Evaluation of four large language models on complex, infectious disease case scenariosSemantic proximity Read

Impact of LLM Assistance on Physician Decision-Making: A Multi-Country Randomized Controlled Trial <sup>∗</sup>Semantic proximity Read

Bias in Large Language Models for Mental Health: Evidence from Vignette-Based Evaluation Across Nine ModelsSemantic proximity Read

Bias in Large Language Models for Mental Health: Evidence from Vignette-Based Evaluation Across Nine ModelsSemantic proximity Read

A Systematic Review of Testing and Evaluation of Healthcare Applications of Large Language Models (LLMs)Semantic proximity Read

Systematic Literature Review of LLM-Large Language Model in Medical: Digital Health, Technology and ApplicationsSemantic proximity Read

How Good Are Large Language Models at Supporting Frontline Healthcare Workers in Low-Resource Settings – A Benchmarking Study & DatasetSemantic proximity Read

General-Purpose vs. Domain-Specific Large Language Models in Antibiotic Clinical Decision-Making: A Double-Blind Evaluation with a 2×2 Factorial DesignSemantic proximity Read

Clinical Safety of AI-Generated Antibiotic Prescribing Advice: Guideline Adherence and Misinformation Risk Among Large Language ModelsSemantic proximity Read

Benchmarking the Confidence of Large Language Models in Clinical QuestionsSemantic proximity Read

Beyond Metrics to Methods: A Scoping Review of Large Language Models for Detection of Social Drivers of Health in Clinical NotesSemantic proximity Read