Article

Thinking ahead: spontaneous prediction in context as a keystone of language in humans and machines

2020-12-03

Abstract excerpt

Departing from traditional linguistic models, advances in deep learning have resulted in a new type of predictive (autoregressive) deep language models (DLMs). Using a self-supervised next-word prediction task, these models are trained to generate appropriate linguistic responses in a given context. We provide empirical evidence that the human brain and autoregressive DLMs share three fundamental computational pri...

Topics

Open a Topic to create a Post that cites this publication.

Uncategorized literature

Identifiers and source

Literature Corpus work: 8f8eac42-26d5-560f-a00d-fa3c68c21fb2
DOI: 10.1101/2020.12.02.403477

Open publication

Related research

Semantic proximity does not establish scientific evidence.

Click a neighbor to travelStep 1 · 12 closest

Thinking ahead: spontaneous prediction in context as a keystone of language in humans and machinesDOI 10.1101/2020.12.02.403477Read

Decoding semantic predictions from EEG prior to word onsetSemantic proximity Read

A note on co-occurrence, transitional probability, and causal inferenceSemantic proximity Read

Gated recurrence enables simple and accurate sequence prediction in stochastic, changing, and structured environmentsSemantic proximity Read

A unifying computational account of temporal context effects in language across the human cortexSemantic proximity Read

The neural architecture of language: Integrative modeling converges on predictive processingSemantic proximity Read

Language processing in brains and deep neural networks: computational convergence and its limitsSemantic proximity Read

A deep hierarchy of predictions enables assignment of semantic roles in online speech comprehensionSemantic proximity Read

Language experience shapes predictive coding of rhythmic sound sequencesSemantic proximity Read

Sensory context improves language prediction in humans and LLMsSemantic proximity Read

Multimodality and Attention Increase Alignment in Natural Language Prediction Between Humans and Computational ModelsSemantic proximity Read

Driving and suppressing the human language network using large language modelsSemantic proximity Read

Sensory context as a universal principle of language in humans and LLMsSemantic proximity Read