Article

Multimodality and Attention Increase Alignment in Natural Language Prediction Between Humans and Computational Models

2024-04-03

Abstract excerpt

<title>Abstract</title> <p>The potential of multimodal generative artificial intelligence (mAI) to replicate human grounded language understanding, including the pragmatic, context-rich aspects of communication, remains to be clarified. Humans are known to use salient multimodal features, such as visual cues, to facilitate the processing of upcoming words. Correspondingly, multimodal computational models can inte...

Topics

Open a Topic to create a Post that cites this publication.

Identifiers and source

Literature Corpus work: af1fb963-75c8-517b-878b-3a2c10893904
DOI: 10.21203/rs.3.rs-3913308/v1

Open publication

Related research

Semantic proximity does not establish scientific evidence.

Click a neighbor to travelStep 1 · 12 closest

Multimodality and Attention Increase Alignment in Natural Language Prediction Between Humans and Computational ModelsDOI 10.21203/rs.3.rs-3913308/v1Read

Sensory context improves language prediction in humans and LLMsSemantic proximity Read

Sensory context as a universal principle of language in humans and LLMsSemantic proximity Read

A Review of Multimodal Vision-Language Models: Foundations, Applications, and Future DirectionsSemantic proximity Read

A Review of Multimodal Vision–Language Models: Foundations, Applications, and Future DirectionsSemantic proximity Read

The neural architecture of language: Integrative modeling converges on predictive processingSemantic proximity Read

Centaur: a foundation model of human cognitionSemantic proximity Read

Thinking ahead: spontaneous prediction in context as a keystone of language in humans and machinesSemantic proximity Read

Active inference, selective attention, and the cocktail party problemSemantic proximity Read

A deep hierarchy of predictions enables assignment of semantic roles in online speech comprehensionSemantic proximity Read

Comprehensive Neural Representations of Naturalistic Stimuli through Multimodal Deep LearningSemantic proximity Read

Hierarchical Semantic Encoding for Contextual Understanding in Large Language ModelsSemantic proximity Read