Article

Upscaling A Smaller LLM to More Parameters via Manual Regressive Distillation

2024-07-23

Abstract excerpt

<title>Abstract</title> <p>The rapid expansion of data and the increasing complexity of linguistic tasks necessitate the development of more powerful language models capable of understanding and generating human language with greater accuracy. The novel approach of scaling GPT-Neo to 70 billion parameters through manual regressive distillation presents a significant advancement, leveraging a systematic knowledge...

Topics

Open a Topic to create a Post that cites this publication.

Identifiers and source

Literature Corpus work: bdccf5ed-d3c5-5eb3-afa5-2f934d74e0ae
DOI: 10.21203/rs.3.rs-4775436/v1

Open publication

Related research

Semantic proximity does not establish scientific evidence.

Click a neighbor to travelStep 1 · 12 closest

Upscaling A Smaller LLM to More Parameters via Manual Regressive DistillationDOI 10.21203/rs.3.rs-4775436/v1Read

Boost Large Language Model Performance through Self-Training with Reward Guided Tree SearchSemantic proximity Read

Large Language Models Robustness Against PerturbationSemantic proximity Read

Unified Efficient Fine-Tuning Techniques for Open-Source Large Language ModelsSemantic proximity Read

Optimizing Large Language Model Scaling with Micro Batch Pipeline and Inference ParallelismSemantic proximity Read

Bridging Finance and AI: A Comprehensive Survey of Large Language Models in Financial ApplicationsSemantic proximity Read

SuperGLUE: The AI raceSemantic proximity Read

HyperLLM: The Next Generation of Large Language Models with Multimodal CapabilitiesSemantic proximity Read

Enhancing Contextual Understanding in Large Language Models with Dynamic Dependency Structures: A Methodological ApproachSemantic proximity Read

Parameter-Efficient Fine-Tuning in Large Models: A Survey of MethodologiesSemantic proximity Read

Larger models yield better results? Streamlined severity classification of ADHD-related concerns using BERT-based knowledge distillationSemantic proximity Read

Enhancements to Large Language Models: Introducing Dynamic Syntactic Insertion for Improved Model Robustness and GeneralizationSemantic proximity Read

Adapting Large Language Models for Low-Resource Regulated Domains: A Fine-Tuning and Retrieval-Augmented Generation Approach to Insurance Information Delivery in KenyaSemantic proximity Read