Article

Layer-wise Adaptive Sparse Fine-Tuning: Boosting LLM Efficiency through Dynamic Ranks and Sparsity

2026-03-16

Abstract excerpt

Large Language Models (LLMs) demand extensive computational resources for fine-tuning, posing significant challenges for widespread customization. While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA mitigate this, their reliance on fixed ranks often leads to inefficient resource allocation and performance bottlenecks due to the heterogeneous importance of different model layers. To address this, we prop...

Topics

Open a Topic to create a Post that cites this publication.

Identifiers and source

Literature Corpus work: 2d00e65b-1b74-58a2-be74-610a49bad2a2
DOI: 10.22541/au.177368931.18403798/v1

Open publication

Related research

Semantic proximity does not establish scientific evidence.

Click a neighbor to travelStep 1 · 12 closest

Layer-wise Adaptive Sparse Fine-Tuning: Boosting LLM Efficiency through Dynamic Ranks and SparsityDOI 10.22541/au.177368931.18403798/v1Read

Unified Efficient Fine-Tuning Techniques for Open-Source Large Language ModelsSemantic proximity Read

Parameter-Efficient Fine-Tuning in Large Models: A Survey of MethodologiesSemantic proximity Read

Upscaling A Smaller LLM to More Parameters via Manual Regressive DistillationSemantic proximity Read

A Survey of Large Language Models: Evolution, Architectures, Adaptation, Benchmarking, Applications, Challenges, and Societal ImplicationsSemantic proximity Read

Efficient Program Optimization through Knowledge-Enhanced LoRA Fine-Tuning of Large Language ModelsSemantic proximity Read

A Comparative Survey of Large Language Models: Foundation, Instruction-Tuned, and Multimodal VariantsSemantic proximity Read

SimLoRA for Enhanced Domain Specific LLM Fine Tuning Using Attention Weighted Low Rank Adaptation and RLHFSemantic proximity Read

Dynamic Contextual Interpolation for More Efficient Model Generalization in Large Language ModelsSemantic proximity Read

Dynamic Token Clustering: A Novel Architectural Enhancement for Large Language ModelsSemantic proximity Read

Engineering A Large Language Model From ScratchSemantic proximity Read

Optimizing Human-Controlled Preference Alignment in Large Language Models via Dense Token Masking: A Methodological ApproachSemantic proximity Read

Adapting Large Language Models for Low-Resource Regulated Domains: A Fine-Tuning and Retrieval-Augmented Generation Approach to Insurance Information Delivery in KenyaSemantic proximity Read