Awesome LLM pruning papers all-in-one repository with integrating all useful resources and insights.
180
53 commits
updated Sep 9, 2026
README
Awesome LLMs Pruning
Integrating useful resources into one repository for large models pruning papers, including one sentence take-away summary, explanation notes such as paper's challenges, blogs or videos, paper tags, source code links and venue.
Please feel free to pull requests or open an issue to add papers.
:fire: Keep updating... Please star it if you find it helpful:)
Table of Contents
Tags of Pruning
Click on the badge, such as 
 |  |  | |  |  |
|---|---|---|---|---|---|
Unstructured | Magnitude | Sparsity e.g. layer or global | Data-free | Without | Frozen |
Structured e.g. Channel, Layer/Depth | Taylor e.g. Hessian | FLOPs | Calibration | Efficient e.g. LoRA | Update |
Semi-structured | Fisher | Latency | Small | Extensive | - |
Other | Trainable | Energy | Medium | Scratch | - |
- | Other | Other | Large | Other | - |
2026
Title & Take-away | Categorization | Note | Code |
|---|---|---|---|
  Less is MoE: Trimming Experts in Domain-Specialist Language Models Traces the failure of prior MoE compression on general-purpose benchmarks to the granularity of compression: capability is distributed across experts but concentrated in a tiny set of FFN sparse intermediate dimensions (on Qwen1.5-MoE, removing 12 of 1.35M routed-FFN dims collapses GSM8K while factual knowledge largely survives). Fisher-MoE therefore prunes within the expert FFN by Fisher importance (outperforming activation-, router-score- and magnitude-based scores) instead of dropping or merging whole experts (expert merge). At 50% MoE compression: capability preserved, ~45% less weight memory, +21% throughput. Qwen1.5/Qwen3-MoE, OLMoE. |       | - | PyTorch |
 SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training One-shot or progressive structured prune of a pretrained MoE on depth (last 25% layers), width (activation-ranked dims), and experts (expert merge/prune with partial-preservation; intra-MoE, not mergekit cross-checkpoint fusion). Pruned init beats train-from-scratch; ~400B continual pretraining with NTP-KD+LM loss and MTP distillation for recovery (merge > prune; progressive > one-shot). Qwen3-Next-80A3B→23A2B (~4×). Cf. Minitron. |      | Challenge | - |
2025
Title & Take-away | Categorization | Note | Code |
|---|---|---|---|
  Fragile Knowledge, Robust Instruction-Following: The Width Pruning Dichotomy in Llama-3.2 PPM (peak-to-peak magnitude) guided GLU-MLP width pruning reveals a fragile/robust capability dichotomy that suggests neuron-level functional specialization, framing the importance criterion as a lever for selective behavioral modification rather than only uniform compression. |     | - | PyTorch |
  ShortGPT: Layers in Large Language Models are More Redundant Than You Expect Delete certain layers, i.e., transformer blocks (given one block consists of both an Attention and an MLP) in LLMs based on Block Influence (BI) score, a novel metric designed to assess the hidden states transformation of each layer. Layers in LLMs could be more redundant than expected. |     | Challenge | PyTorch |
 Puzzle: Distillation-Based NAS for Inference-Optimized LLMs Hardware-aware NAS framework combining channel and depth pruning via: (1) attention subblock pruning - reduce multi-heads to 8/4/2/1 or remove entirely with no-op, and (2) MLP subblock pruning - reduce channels to 87%/75%/50%/25%/20%/10% or remove with no-op. Achieves 2.17x speedup on Llama-70B→Nemotron-51B/49B with 98.4% accuracy using 45B tokens. |     | Challenge | - |
  2SSP: A Two-Stage Framework for Structured Pruning of LLMs Two-stage structured pruning framework combining width and depth pruning: Stage 1 (Width Pruning) removes entire neurons and their corresponding rows/columns in feed-forward networks yet MLP; Stage 2 (Depth Pruning) removes entire Attention submodules. Novel mechanism to balance sparsity rate between stages. With no retraining. |   | - | PyTorch |
  The Unreasonable Ineffectiveness of the Deeper Layers A simple layer/depth pruning to remove n consecutive or contiguous layers from popular families of open-weight pretrained LLMs by minimizing the angular distance between layers' representations. Parameter-efficient finetuning method is applied to further reduce computational resources of finetuning. | | Challenge | PyTorch |
 Uncovering the Redundancy in Transformers via a Unified Study of Layer Dropping Explore the redundancy in three key Transformer components: Block, MLP, and Attention, where one Block = Attention + MLP (see ShortGPT). "Block/MLP drop" leads to significant performance degradation. A fine-grained "Attention Drop" has minimal impact on model accuracy, and alleviates memory overhead due to KV cache. Similarity-based metric to evaluate component's importance. | | Reviews | PyTorch |
| Rethinking the Impact of Heterogeneous Sublayers in Transformers Instead of pruning entire coarse-grained transformer blocks, this paper proposed a finer granularity depth pruning method that prunes sublayers with treating single transformer block as 2 sublayers, i.e., Multi-Head Attention (MHA) and MLP. |  | - | - |
 Streamlining Redundant Layers to Compress Large Language Models LLM-Streamline comprises two components: layer pruning and layer replacement. First, certain contiguous redundant layers are pruned from the LLMs based on cosine similarity importance metric; Then, a lightweight network is trained on a small subset of SlimPajama to replace the pruned layers to restore the model’s performance. |   | Challenge Reviews | PyTorch |
 Reassessing Layer Pruning in LLMs: New Insights and Methods Validate seven different layer selection metrics including Random, Reverse-order, Magnitude, Taylor, Perplexity and Cosine Similarity (BI). Reverse-order pruning is simple yet effective. LoRA performs worse than a simple partial-layer fine-tuning. Iterative pruning offers no benefit compared to one-shot pruning. |    | Challenge Reviews | PyTorch |
 You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning A novel policy learning approach for model compression that eliminates the need for calibration data. The method learns to make pruning decisions through reinforcement learning, optimizing for both performance and efficiency without requiring fine-tuning datasets. |   | Reviews | PyTorch |
 Prompt-based Depth Pruning of Large Language Models A dynamic depth pruning method that trains a lightweight router to predict which transformer layers to omit based on input prompts. The method first generates candidate omission sets using task likelihood loss on calibration data, then trains a BERT-based router to select the optimal layer combination for each prompt, achieving better task-specific performance than static pruning approaches. |  | Reviews | PyTorch |
 DLP: Dynamic Layerwise Pruning in Large Language Models A dynamic layerwise pruning method that adaptively determines the relative importance of each layer by integrating model weights with input activation information, assigning pruning rates accordingly. Uses median-based layer importance scoring to avoid hyperparameter tuning, achieving superior performance at high sparsity levels compared to uniform layerwise pruning approaches like OWL. |    | Reviews | PyTorch |
| SlimLLM: Accurate Structured Pruning for Large Language Models An effective structured pruning method that evaluates channel and attention head importance holistically rather than aggregating individual elements. Uses Pearson similarity-based head pruning with greedy search, PCA-guided FFN channel pruning, and lightweight linear regression for performance recovery. Achieves state-of-the-art results with 3.4× speedup on LLaMA-7B while maintaining competitive accuracy through layer-wise dynamic sparsity allocation. |   | Reviews | - |
  MDP: Multidimensional Vision Model Pruning with Latency Constraint Multi-Dimensional Pruning (MDP) jointly optimizes across various pruning granularities including channels, query, key, heads, embeddings, and blocks. Uses advanced latency modeling to balance latency and accuracy by formulating pruning as a Mixed-Integer Nonlinear Program (MINLP). Supports both CNNs and transformers. Achieves 28% speed increase with +1.4 Top-1 accuracy improvement over HALP for ResNet50, and 37% additional acceleration with +0.7 Top-1 accuracy improvement over Isomorphic transformer pruning. |    | - | PyTorch |
 OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition Extends LoSparse sparse+low-rank decomposition with outlier-aware scaling using input embedding second moments to preserve critical outlier features in transformers. Uses alternating thresholding algorithm for robust decomposition without retraining. Achieves up to 60% compression on LLMs (Llama-3, Phi-3) and vision transformers (ViT, DINOv2) with 1.37× CPU inference speedup over prior methods including LoSparse. |   | Reviews | PyTorch |
| LLaMaFlex: Many-in-one LLMs via Generalized Pruning and Weight Sharing A novel nested weight-shared architecture that can be pruned across both width and depth dimensions in zero-shot manner to instantly yield compressed models. Uses end-to-end Gumbel Softmax-based router for architecture selection and policy-aware modulation for enhanced expressiveness. Enables "train once, deploy many" paradigm with single continued training phase (~60B tokens) on pretrained model. Tested on Llama 3.1 8B, produces family of compressed models achieving accuracy on par with or better than state-of-the-art approaches. |    | Reviews | - |
2024
Title & Take-away | Categorization | Note | Code |
|---|---|---|---|
  Shortened LLaMA: A Simple Depth Pruning for Large Language Models First identify unimportant Transformer blocks (bigger and coarse units), then perform one-shot pruning with Perplexity (PPL) as pruning criteria and light LoRA retraining. Show fast inference and good zero-shot capabilities. |    | Challenge | PyTorch |
  A deeper look at depth pruning of LLMs This work explores different block importance metrics including cosine similarity, relativeL1/L2 and Shapleyvalue-based, to take a deeper look at depth pruning of LLMs. Further exam the impact of droping individual Attention and MLP layers. Two simple performance recovery techniques are applied on calibration dataset. |  | - | PyTorch |
  Compact Language Models via Pruning and Knowledge Distillation Prune LLMs structurally along different axes such as layer, neuron, head, and embedding channel, similar to NAS that searches over different dimensions. Difference lies in the defined search space that for pruning a pre-trained large model as search space (simpler) while NAS searches over a manually-pre-defined search space (more complex) from scratch. Different proxy importance scores are estimated for depth and width pruning. Retraining with knowledge distillation requires up to 40x fewer training tokens. |   | Challenge Blog | PyTorch |
  Keyformer: KV Cache reduction through attention sparsification for Efficient Generative Inference Keyformer, a successor to H2O (see below), uses a Gumbel softmax-based score function instead of solely attention scores in H2O, for dynamically identifying and retaining top-k key tokens, to reduce KV cache size. A sliding window drawn from Sparse Transformer is used to retain (not prune) w recent representative tokens, yileding a mixture of recent and key tokens. |    | Challenge Blog Summary | PyTorch |
  A Simple and Effective Pruning Approach for Large Language Models A pruning metric termed Wanda that considers both weight magnitudes and input activation norms to prune weights per-output basis instead of layer-wise, requiring no retraining or weight update. A simplified version of SparseGPT. |   | Challenge Blog Reviews | PyTorch |
| The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits 1.58 bits to quantize every single parameter of the LLM in ternary -1, 0, or +1. This can be viewed as an 1-bit binarization -1 or 1 along with unstructured pruning 0. It matches the full-precision Transformer LLM with the same model size and training tokens when trained from scratch, with 1.58-bit weights and 8-bit activations. |   | Challenge Discussion | PyTorch |
 BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation Adaptively allocate optimal sparsity ratio of each layer within a transformer block by minizming block-wise reconstruction error. To do so, a parameter-efficient algorithm is developed with ony optimizing few learnable coefficients e.g., 100. Pre-trained weights are frozen. |  | Challenge Reviews | PyTorch |
 Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning In first stage, training-aware pruning learns masks satisfying specified target by imposing regularization on ~0.4B tokens; then retrain on other ~5B tokens of RedPajama dataset. Dynamic batch loading method to update the composition of sampled data per mini-batch across different domains. |   | Challenge Blog Reviews | PyTorch |
  Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity Allocate non-uniform sparsity ratios across different layers guided by the principle that a layer with higher proportion of outliers should have a lower sparsity, then apply the more tailored layer-wise sparsity directly into Wanda and SparseGPT. |  | Challenge Reviews | PyTorch |
 Plug-and-Play: An Efficient Post-training Pruning Method for Large Language Models Propose a new pruning criteria named RIA for LLMs. In N:M structures, introduce a column permutation matrix for score matrix to maximize the total retained weight importance. No retraining. |  | Challenge Reviews | PyTorch |
| Compresso: Structured Pruning with Collaborative Prompting Learns Compact Large Language Models Retrain LLMs' weights with lightweight LoRA, and optimize structured-pruning masks with efficient trainable parameters in differentiable way on instruction-tuning Alpaca dataset. Collaborative prompt is used to help pruning task. |   | Challenge Reviews | PyTorch |
| Scaling Laws for Sparsely-Connected Foundation Models Discover scaling law of weight sparsity, formulating the scaling relationships between weight sparsity, non-zero parameter numbers, and training data size. Revealing an increasing optimal sparsity with more training data and offering insights for improved computational efficiency. |  | Challenge Reviews | - |
 The LLM Surgeon This paper introduces LLM Surgeon, a method that enhances the efficiency of second-order Hessian-based pruning techniques, such as Optimal Brain Surgeon, by employing Kronecker-factored approximations of the Fisher information matrix. The approach establishes closed-form solutions. Prune OPT models and Llamav2-7B by 20%-30% achieves a negligible loss in performance. |    | Challenge Reviews | PyTorch |
 Accurate Retraining-free Pruning for Pretrained Encoder-based Language Models Retraining-free pruning for encoder-based language model such as BERT to preserve the knowledge of PLMs through sublayer-wise iterative pruning, from the bottom to the top sublayer. |    | Challenge Reviews | PyTorch |
 Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs Dynamic Sparse No Training (DSNT) involves iterative pruning-and-growing steps that only updating sparse mask yet mask adaptation by minimizing reconstruction error e.g. proxy of perplexity; Enable a higher 60% or 70% sparsity rate; Training-free. |  | Challenge Reviews | PyTorch |
  Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity An effective software framework for tensor cores (do not allow skipping arbitrary element-level computations) based unstructured SpMM, leveraging on-chip resources for efficient sparse data extraction and computation/memory-access overlapping. Improving memory bandwidth utilization in GPU. |    | - | Python/C++ |
2023
Title & Take-away | Categorization | Note | Code |
|---|---|---|---|
  H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models During prompt and generation phase, dynamically prune the unimportant tokens based on accumulated attention scores, yet maintaining a constant small Key-Value Cache (KV cache ) size with k tokens. | | Challenge | PyTorch |
  SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot Post-training method for pruning LLMs in one-shot without any retraining. Updating weights by solving a layer-wise weight reconstruction problem. |  | Challenge Blog | PyTorch |
 LLM-Pruner: On the Structural Pruning of Large Language Models First discover all coupled structures following Depgraph, then estimate grouped importance of coupled structure on calibration, then prune less important groups, and last finetune with efficient LoRA on Alpaca dataset consists of 50K instruction-response pairs. | | Challenge | PyTorch |
 LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation Novel model compression technique that approximates weight matrices as the sum of a low-rank matrix and a sparse matrix. Combines advantages of both low-rank approximations and pruning while avoiding their limitations. Low-rank approximation compresses coherent and expressive parts in neurons, while pruning removes incoherent and non-expressive parts. Pruning enhances diversity of low-rank approximations, and low-rank approximation prevents pruning from losing expressive neurons. |  | - | PyTorch |
 The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter Revisiting magnitude pruning and several interesting findings on pruning large scaled models. Most performances are reported with fine-tuned downstream tasks, except for that on modern-scale LLMs where no retraining is performed. |    | Challenge | PyTorch |
Contributors
Similar repos
Awesome LLM pruning papers all-in-one repository with integrating all useful resources and insights.
180
53 commits
updated Sep 9, 2026
README
Awesome LLMs Pruning
Integrating useful resources into one repository for large models pruning papers, including one sentence take-away summary, explanation notes such as paper's challenges, blogs or videos, paper tags, source code links and venue.
Please feel free to pull requests or open an issue to add papers.
:fire: Keep updating... Please star it if you find it helpful:)
Table of Contents
Tags of Pruning
Click on the badge, such as
| | | | | |
|---|---|---|---|---|---|
Unstructured | Magnitude | Sparsity e.g. layer or global | Data-free | Without | Frozen |
Structured e.g. Channel, Layer/Depth | Taylor e.g. Hessian | FLOPs | Calibration | Efficient e.g. LoRA | Update |
Semi-structured | Fisher | Latency | Small | Extensive | - |
Other | Trainable | Energy | Medium | Scratch | - |
- | Other | Other | Large | Other | - |
2026
Title & Take-away | Categorization | Note | Code |
|---|---|---|---|
| Less is MoE: Trimming Experts in Domain-Specialist Language Models Traces the failure of prior MoE compression on general-purpose benchmarks to the granularity of compression: capability is distributed across experts but concentrated in a tiny set of FFN sparse intermediate dimensions (on Qwen1.5-MoE, removing 12 of 1.35M routed-FFN dims collapses GSM8K while factual knowledge largely survives). Fisher-MoE therefore prunes within the expert FFN by Fisher importance (outperforming activation-, router-score- and magnitude-based scores) instead of dropping or merging whole experts (expert merge). At 50% MoE compression: capability preserved, ~45% less weight memory, +21% throughput. Qwen1.5/Qwen3-MoE, OLMoE. | | - | PyTorch |
| SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training One-shot or progressive structured prune of a pretrained MoE on depth (last 25% layers), width (activation-ranked dims), and experts (expert merge/prune with partial-preservation; intra-MoE, not mergekit cross-checkpoint fusion). Pruned init beats train-from-scratch; ~400B continual pretraining with NTP-KD+LM loss and MTP distillation for recovery (merge > prune; progressive > one-shot). Qwen3-Next-80A3B→23A2B (~4×). Cf. Minitron. | | Challenge | - |
2025
Title & Take-away | Categorization | Note | Code |
|---|---|---|---|
| Fragile Knowledge, Robust Instruction-Following: The Width Pruning Dichotomy in Llama-3.2 PPM (peak-to-peak magnitude) guided GLU-MLP width pruning reveals a fragile/robust capability dichotomy that suggests neuron-level functional specialization, framing the importance criterion as a lever for selective behavioral modification rather than only uniform compression. | | - | PyTorch |
| ShortGPT: Layers in Large Language Models are More Redundant Than You Expect Delete certain layers, i.e., transformer blocks (given one block consists of both an Attention and an MLP) in LLMs based on Block Influence (BI) score, a novel metric designed to assess the hidden states transformation of each layer. Layers in LLMs could be more redundant than expected. | | Challenge | PyTorch |
| Puzzle: Distillation-Based NAS for Inference-Optimized LLMs Hardware-aware NAS framework combining channel and depth pruning via: (1) attention subblock pruning - reduce multi-heads to 8/4/2/1 or remove entirely with no-op, and (2) MLP subblock pruning - reduce channels to 87%/75%/50%/25%/20%/10% or remove with no-op. Achieves 2.17x speedup on Llama-70B→Nemotron-51B/49B with 98.4% accuracy using 45B tokens. | | Challenge | - |
| 2SSP: A Two-Stage Framework for Structured Pruning of LLMs Two-stage structured pruning framework combining width and depth pruning: Stage 1 (Width Pruning) removes entire neurons and their corresponding rows/columns in feed-forward networks yet MLP; Stage 2 (Depth Pruning) removes entire Attention submodules. Novel mechanism to balance sparsity rate between stages. With no retraining. | | - | PyTorch |
| The Unreasonable Ineffectiveness of the Deeper Layers A simple layer/depth pruning to remove n consecutive or contiguous layers from popular families of open-weight pretrained LLMs by minimizing the angular distance between layers' representations. Parameter-efficient finetuning method is applied to further reduce computational resources of finetuning. | | Challenge | PyTorch |
| Uncovering the Redundancy in Transformers via a Unified Study of Layer Dropping Explore the redundancy in three key Transformer components: Block, MLP, and Attention, where one Block = Attention + MLP (see ShortGPT). "Block/MLP drop" leads to significant performance degradation. A fine-grained "Attention Drop" has minimal impact on model accuracy, and alleviates memory overhead due to KV cache. Similarity-based metric to evaluate component's importance. | | Reviews | PyTorch |
| Rethinking the Impact of Heterogeneous Sublayers in Transformers Instead of pruning entire coarse-grained transformer blocks, this paper proposed a finer granularity depth pruning method that prunes sublayers with treating single transformer block as 2 sublayers, i.e., Multi-Head Attention (MHA) and MLP. | | - | - |
| Streamlining Redundant Layers to Compress Large Language Models LLM-Streamline comprises two components: layer pruning and layer replacement. First, certain contiguous redundant layers are pruned from the LLMs based on cosine similarity importance metric; Then, a lightweight network is trained on a small subset of SlimPajama to replace the pruned layers to restore the model’s performance. | | Challenge Reviews | PyTorch |
| Reassessing Layer Pruning in LLMs: New Insights and Methods Validate seven different layer selection metrics including Random, Reverse-order, Magnitude, Taylor, Perplexity and Cosine Similarity (BI). Reverse-order pruning is simple yet effective. LoRA performs worse than a simple partial-layer fine-tuning. Iterative pruning offers no benefit compared to one-shot pruning. | | Challenge Reviews | PyTorch |
| You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning A novel policy learning approach for model compression that eliminates the need for calibration data. The method learns to make pruning decisions through reinforcement learning, optimizing for both performance and efficiency without requiring fine-tuning datasets. | | Reviews | PyTorch |
| Prompt-based Depth Pruning of Large Language Models A dynamic depth pruning method that trains a lightweight router to predict which transformer layers to omit based on input prompts. The method first generates candidate omission sets using task likelihood loss on calibration data, then trains a BERT-based router to select the optimal layer combination for each prompt, achieving better task-specific performance than static pruning approaches. | | Reviews | PyTorch |
| DLP: Dynamic Layerwise Pruning in Large Language Models A dynamic layerwise pruning method that adaptively determines the relative importance of each layer by integrating model weights with input activation information, assigning pruning rates accordingly. Uses median-based layer importance scoring to avoid hyperparameter tuning, achieving superior performance at high sparsity levels compared to uniform layerwise pruning approaches like OWL. | | Reviews | PyTorch |
| SlimLLM: Accurate Structured Pruning for Large Language Models An effective structured pruning method that evaluates channel and attention head importance holistically rather than aggregating individual elements. Uses Pearson similarity-based head pruning with greedy search, PCA-guided FFN channel pruning, and lightweight linear regression for performance recovery. Achieves state-of-the-art results with 3.4× speedup on LLaMA-7B while maintaining competitive accuracy through layer-wise dynamic sparsity allocation. | | Reviews | - |
| MDP: Multidimensional Vision Model Pruning with Latency Constraint Multi-Dimensional Pruning (MDP) jointly optimizes across various pruning granularities including channels, query, key, heads, embeddings, and blocks. Uses advanced latency modeling to balance latency and accuracy by formulating pruning as a Mixed-Integer Nonlinear Program (MINLP). Supports both CNNs and transformers. Achieves 28% speed increase with +1.4 Top-1 accuracy improvement over HALP for ResNet50, and 37% additional acceleration with +0.7 Top-1 accuracy improvement over Isomorphic transformer pruning. | | - | PyTorch |
| OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition Extends LoSparse sparse+low-rank decomposition with outlier-aware scaling using input embedding second moments to preserve critical outlier features in transformers. Uses alternating thresholding algorithm for robust decomposition without retraining. Achieves up to 60% compression on LLMs (Llama-3, Phi-3) and vision transformers (ViT, DINOv2) with 1.37× CPU inference speedup over prior methods including LoSparse. | | Reviews | PyTorch |
| LLaMaFlex: Many-in-one LLMs via Generalized Pruning and Weight Sharing A novel nested weight-shared architecture that can be pruned across both width and depth dimensions in zero-shot manner to instantly yield compressed models. Uses end-to-end Gumbel Softmax-based router for architecture selection and policy-aware modulation for enhanced expressiveness. Enables "train once, deploy many" paradigm with single continued training phase (~60B tokens) on pretrained model. Tested on Llama 3.1 8B, produces family of compressed models achieving accuracy on par with or better than state-of-the-art approaches. | | Reviews | - |
2024
Title & Take-away | Categorization | Note | Code |
|---|---|---|---|
| Shortened LLaMA: A Simple Depth Pruning for Large Language Models First identify unimportant Transformer blocks (bigger and coarse units), then perform one-shot pruning with Perplexity (PPL) as pruning criteria and light LoRA retraining. Show fast inference and good zero-shot capabilities. | | Challenge | PyTorch |
| A deeper look at depth pruning of LLMs This work explores different block importance metrics including cosine similarity, relativeL1/L2 and Shapleyvalue-based, to take a deeper look at depth pruning of LLMs. Further exam the impact of droping individual Attention and MLP layers. Two simple performance recovery techniques are applied on calibration dataset. | | - | PyTorch |
| Compact Language Models via Pruning and Knowledge Distillation Prune LLMs structurally along different axes such as layer, neuron, head, and embedding channel, similar to NAS that searches over different dimensions. Difference lies in the defined search space that for pruning a pre-trained large model as search space (simpler) while NAS searches over a manually-pre-defined search space (more complex) from scratch. Different proxy importance scores are estimated for depth and width pruning. Retraining with knowledge distillation requires up to 40x fewer training tokens. | | Challenge Blog | PyTorch |
| Keyformer: KV Cache reduction through attention sparsification for Efficient Generative Inference Keyformer, a successor to H2O (see below), uses a Gumbel softmax-based score function instead of solely attention scores in H2O, for dynamically identifying and retaining top-k key tokens, to reduce KV cache size. A sliding window drawn from Sparse Transformer is used to retain (not prune) w recent representative tokens, yileding a mixture of recent and key tokens. | | Challenge Blog Summary | PyTorch |
| A Simple and Effective Pruning Approach for Large Language Models A pruning metric termed Wanda that considers both weight magnitudes and input activation norms to prune weights per-output basis instead of layer-wise, requiring no retraining or weight update. A simplified version of SparseGPT. | | Challenge Blog Reviews | PyTorch |
| The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits 1.58 bits to quantize every single parameter of the LLM in ternary -1, 0, or +1. This can be viewed as an 1-bit binarization -1 or 1 along with unstructured pruning 0. It matches the full-precision Transformer LLM with the same model size and training tokens when trained from scratch, with 1.58-bit weights and 8-bit activations. | | Challenge Discussion | PyTorch |
| BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation Adaptively allocate optimal sparsity ratio of each layer within a transformer block by minizming block-wise reconstruction error. To do so, a parameter-efficient algorithm is developed with ony optimizing few learnable coefficients e.g., 100. Pre-trained weights are frozen. | | Challenge Reviews | PyTorch |
| Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning In first stage, training-aware pruning learns masks satisfying specified target by imposing regularization on ~0.4B tokens; then retrain on other ~5B tokens of RedPajama dataset. Dynamic batch loading method to update the composition of sampled data per mini-batch across different domains. | | Challenge Blog Reviews | PyTorch |
| Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity Allocate non-uniform sparsity ratios across different layers guided by the principle that a layer with higher proportion of outliers should have a lower sparsity, then apply the more tailored layer-wise sparsity directly into Wanda and SparseGPT. | | Challenge Reviews | PyTorch |
| Plug-and-Play: An Efficient Post-training Pruning Method for Large Language Models Propose a new pruning criteria named RIA for LLMs. In N:M structures, introduce a column permutation matrix for score matrix to maximize the total retained weight importance. No retraining. | | Challenge Reviews | PyTorch |
| Compresso: Structured Pruning with Collaborative Prompting Learns Compact Large Language Models Retrain LLMs' weights with lightweight LoRA, and optimize structured-pruning masks with efficient trainable parameters in differentiable way on instruction-tuning Alpaca dataset. Collaborative prompt is used to help pruning task. | | Challenge Reviews | PyTorch |
| Scaling Laws for Sparsely-Connected Foundation Models Discover scaling law of weight sparsity, formulating the scaling relationships between weight sparsity, non-zero parameter numbers, and training data size. Revealing an increasing optimal sparsity with more training data and offering insights for improved computational efficiency. | | Challenge Reviews | - |
| The LLM Surgeon This paper introduces LLM Surgeon, a method that enhances the efficiency of second-order Hessian-based pruning techniques, such as Optimal Brain Surgeon, by employing Kronecker-factored approximations of the Fisher information matrix. The approach establishes closed-form solutions. Prune OPT models and Llamav2-7B by 20%-30% achieves a negligible loss in performance. | | Challenge Reviews | PyTorch |
| Accurate Retraining-free Pruning for Pretrained Encoder-based Language Models Retraining-free pruning for encoder-based language model such as BERT to preserve the knowledge of PLMs through sublayer-wise iterative pruning, from the bottom to the top sublayer. | | Challenge Reviews | PyTorch |
| Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs Dynamic Sparse No Training (DSNT) involves iterative pruning-and-growing steps that only updating sparse mask yet mask adaptation by minimizing reconstruction error e.g. proxy of perplexity; Enable a higher 60% or 70% sparsity rate; Training-free. | | Challenge Reviews | PyTorch |
| Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity An effective software framework for tensor cores (do not allow skipping arbitrary element-level computations) based unstructured SpMM, leveraging on-chip resources for efficient sparse data extraction and computation/memory-access overlapping. Improving memory bandwidth utilization in GPU. | | - | Python/C++ |
2023
Title & Take-away | Categorization | Note | Code |
|---|---|---|---|
| H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models During prompt and generation phase, dynamically prune the unimportant tokens based on accumulated attention scores, yet maintaining a constant small Key-Value Cache (KV cache ) size with k tokens. | | Challenge | PyTorch |
| SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot Post-training method for pruning LLMs in one-shot without any retraining. Updating weights by solving a layer-wise weight reconstruction problem. | | Challenge Blog | PyTorch |
| LLM-Pruner: On the Structural Pruning of Large Language Models First discover all coupled structures following Depgraph, then estimate grouped importance of coupled structure on calibration, then prune less important groups, and last finetune with efficient LoRA on Alpaca dataset consists of 50K instruction-response pairs. | | Challenge | PyTorch |
| LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation Novel model compression technique that approximates weight matrices as the sum of a low-rank matrix and a sparse matrix. Combines advantages of both low-rank approximations and pruning while avoiding their limitations. Low-rank approximation compresses coherent and expressive parts in neurons, while pruning removes incoherent and non-expressive parts. Pruning enhances diversity of low-rank approximations, and low-rank approximation prevents pruning from losing expressive neurons. | | - | PyTorch |
| The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter Revisiting magnitude pruning and several interesting findings on pruning large scaled models. Most performances are reported with fine-tuned downstream tasks, except for that on modern-scale LLMs where no retraining is performed. | | Challenge | PyTorch |