Article

Development of a Human Evaluation Framework and Correlation with Automated Metrics for Natural Language Generation of Medical Diagnoses

2024-03-21

Abstract excerpt

In the evolving landscape of clinical Natural Language Generation (NLG), assessing abstractive text quality remains challenging, as existing methods often overlook generative task complexities. This work aimed to examine the current state of automated evaluation metrics in NLG in healthcare. To have a robust and well-validated baseline with which to examine the alignment of these metrics, we created a comprehensiv...

Topics

Open a Topic to create a Post that cites this publication.

Identifiers and source

Literature Corpus work: 6b4f3b98-0182-5f5e-b97f-641c854081b4
DOI: 10.1101/2024.03.20.24304620

Open publication

Related research

Semantic proximity does not establish scientific evidence.

Click a neighbor to travelStep 1 · 12 closest

Development of a Human Evaluation Framework and Correlation with Automated Metrics for Natural Language Generation of Medical DiagnosesDOI 10.1101/2024.03.20.24304620Read

A Clinically-Informed Framework for Evaluating Vision-Language Models in Radiology Report Generation: Taxonomy of Errors and Risk-Aware MetricSemantic proximity Read

Automating Evaluation of AI Text Generation in Healthcare with a Large Language Model (LLM)-as-a-JudgeSemantic proximity Read

Testing and Evaluation of Generative Large Language Models in Electronic Health Record Applications: A Systematic ReviewSemantic proximity Read

A Framework to Assess Clinical Safety and Hallucination Rates of LLMs for Medical Text SummarisationSemantic proximity Read

Protocol For Human Evaluation of Artificial Intelligence Chatbots in Clinical ConsultationsSemantic proximity Read

Faithful AI in Medicine: A Systematic Review with Large Language Models and BeyondSemantic proximity Read

Retrieval Augmented Generation for Large Language Models in Healthcare: A Systematic ReviewSemantic proximity Read

Expert-level validation of AI-generated medical text with scalable language modelsSemantic proximity Read

HumanELY: Human evaluation of LLM yield, using a novel web-based evaluation toolSemantic proximity Read

Measuring the Quality of AI-Generated Clinical Notes: A Systematic Review and Experimental Benchmark of Evaluation MethodsSemantic proximity Read

Mitigating Hallucinations in Large Language Models: A Comparative Study of RAG-enhanced vs. Human-Generated Medical TemplatesSemantic proximity Read

MedReadCtrl: Personalizing medical text generation with readability-controlled instruction learningSemantic proximity Read