Nonequilibrium training dynamics and scaling laws of language models

Abstract excerpt

<title>Abstract</title> <p>Large language models (LLMs) exhibit empirical scaling laws, yet the mechanistic origin of these power laws remains poorly understood. We propose a stochastic multiscale theory of LLM training dynamics, grounded in a formal analogy with nonequilibrium systems such as turbulence and cosmic structure formation, where scaling laws emerge naturally from scale-space transport processes. Cent...

Topics

Open a Topic to create a Post that cites this publication.

• Adaptation, Biological
• Biological Evolution
• Genetics, Population
• Models, Theoretical
• Phenotype
• Population Dynamics
• Selection, Genetic
• Species Specificity

Identifiers and source

Literature Corpus work

fdd05eb0-6f80-5b04-b4f4-e6e4eb7c738c

DOI

10.21203/rs.3.rs-10529159/v1