Shadowgraph
TrendingAbout
Sign in

hrcheng1066/awesome-pruning

309

340 commits

updated Aug 20, 2024

See the code

README

awesome-pruning

PRs WelcomeAwesome

Table of Contents

0. Overview

The repo includes the ongoing updates of representative neural network pruning papers and open-source codes.
Our paper [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations] (Paper Link), accepted by TPAMI 2024.

Taxonomy: In our survey, we provide a comprehensive review of the state-of-the-art in deep neural network pruning, which we categorize along five orthogonal axes: Universal/Specific Speedup, When to Prune, Pruning Criteria, Learn to Prune, and Fusion of Pruning and Other Techniques.

1. When to Prune

Type Explanation

TypeLFCNHBMEWPOther
ExplanationLayer pruningFilter pruningChannel pruningNeuron pruningHead pruningBlock pruningMatrix pruningEmbedding pruningWeight pruningPioneer workother types

1.1 Static Pruning

1.1.1 Pruning Before Training

1.1.1.1 Pruning CNNs
Pruning Before Training CNNs 2024
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01No Free Prune: Information-Theoretic Barriers to Pruning at InitializationICMLW--Image Classification2024
Pruning Before Training CNNs 2023
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Data-Free Model Pruning at Initialization via ExpandersCVPRWWRRegPyTorch(Author)Image Classification2023
02Revisiting Pruning as Initialization through the Lens of Ramanujan GraphICLR (TOP 5%)W-PyTorch(Author)Image Classification2023
03Pruning at Initialization - A Sketching PerspectivearXivW--Image Classification2023
04NTK-SAP: Improving neural network pruning by aligning training dynamicsICLRWNTK-SAPPyTorch(Author)Image Classification2023
Pruning Before Training CNNs 2022
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Prospect Pruning: Finding Trainable Weights at Initialization using Meta-GradientsICLRWFProsPrPyTorch(Author)Image Classification2022
02Dual Lottery Ticket HypothesisICLRWRSTPyTorch(Author)Image Classification2022
03Recent Advances on Neural Network Pruning at InitializationIJCAIW-PyTorch(Author)Image Classification2022
04The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse TrainingICLRW-PyTorch(Author)Image Classification2022
05Structured Pruning is All You Need for Pruning CNNs at InitializationarXivCPreCropping-Image Classification2022
Pruning Before Training CNNs 2021
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Progressive Skeletonization: Trimming More Fat from a network at initializationICLRWFORCEPyTorch(Author)Image Classification2021
02Robust Pruning at InitializationICLRWSPB-Image Classification2021
03A Unified Paths Perspective for Pruning at InitializationarXivW--Image Classification2021
04Prunining Neural Networks at Initialization: Why are We Missing the Mark?ICLRW--Image Classification2021
05Why is Pruning at Initialization Immune to Reinitializating and Shuffling?)arXivW--Image Classification2021
06Dense for the Price of Sparse: Improved Performance of Sparsely Initialized Networks via a Subspace Offset)ICMLWDCTpSPyTorch(Author)Image Classification2021
Pruning Before Training CNNs 2020
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Single Shot Structured Pruning Before TrainingarXivC3SPPyTorch(Author)Image Classification2020
02A Signal Propagation Perspective for Pruning Neural Networks at InitializationICLR (Spotlight)W-TensorFLow(Author)Image Classification2020
03Picking Winning Tickets before Training by Preserving Gradient Flow)ICLRWGraSPPyTorch(Author)Image Classification2020
04Pruning from ScratchAAAIC-PyTorch(Author)Image Classification2020
05Pruning neural networks without any data by iteratively conserving synaptic flowNeurIPSWSynFlowPyTorch(Author)Image Classification2020
06Sanity-Checking Pruning Methods: Random Tickets can Win the JackpotNeurIPSWSmart-RatiosPyTorch(Author)Image Classification2020
07Prunining via Iterative Ranking of Sensitivity StaticsarXivWFCSNIP-itPyTorch(Author)Image Classification2020
08What’s Hidden in a Randomly Weighted Neural Network?CVPRW-PyTorch(Author)Image Classification2020
09Finding trainable sparse networks through Neural Tangent TransferICMLW-PyTorch(Author)Image Classification2020
Pruning Before Training CNNs 2019
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01SNIP: Single-shot Network Pruning based on Connection SensitivityICLRWPSNIPTensorFLow(Author)Image Classification2019

1.1.2 Pruning During Training

1.1.2.1 Pruning CNNs
Pruning During Training CNNs 2024
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Auto-Train-Once: Controller Network Guided Automatic Network Pruning from ScratchCVPRWATOPyTorch(Author)Image Classification2024
Pruning During Training CNNs 2023
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01PDP: Parameter-free Differentiable Pruning is All You NeedNeurIPSWC--Vision&NLP2023
02LAPP: Layer Adaptive Progressive Pruning for Compressing CNNs from ScratcharXivFLAPP-Image Classification2023
Pruning During Training CNNs 2022
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01SuperTickets: Drawing Task-Agnostic Lottery Tickets from Supernets via Jointly Architecture Searching and Parameter PruningECCVWSuperTicketsPyTorch(Author)Image Classification&Object Detection&Human Pose Estimation2022
02Deep ensembling with no overhead for either training or testing: The all-round blessings of dynamic sparsityICLRWFreeTicketsPyTorch(Anthor)Image Classification2022
03Gradient Flow in Sparse Neural Networks and How Lottery Tickets WinAAAIW-PyTorch(Anthor)Image Classification2022
Pruning During Training CNNs 2021
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Training Adversarially Robust Sparse Networks via Bayesian Connectivity SamplingICMLW-PyTorch(Anthor)Adversarial Robustness2021
02Training Neural Networks with Fixed Sparse MasksNeurIPSW-PyTorch(Author)Image Classification2021
03DPFPS: Dynamic and Progressive Filter Pruning for Compressing Convolutional Neural Networks from ScratchAAAICDPFPSPyTorch(Author)Image Classification2021
04Sparse Training via Boosting Pruning Plasticity with NeuroregenerationNeurIPSWFGraNetPyTorch(Author)Image Classification2021
05Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse TrainingICMLWITOPPyTorch(Anthor)Image Classification2021
06Dense for the Price of Sparse: Improved Performance of Sparsely Initialized Networks via a Subspace OffsetICMLWDCTpSPyTorch(Anthor)Image Classification2021
Pruning During Training CNNs 2020
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Pruning Filter in FilterNeurIPSOtherSWPPyTorch(Author)Image Classification2020
02Dynamic Sparse Training: Find Effective Sparse Network from Scratch with Trainable Masked LayersICLRNFDSTPyTorch(Author)Image Classification2020
03DSA: More Efficient Budgeted Pruning via Differentiable Sparsity AllocationECCVFDSAPyTorch(Author)Image Classification2020
04Dynamic Model Pruning with FeedbackICLRWFDPFPyTorch(3rd)Image Classification2020
Pruning During Training CNNs 2019
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Filter Pruning via Geometric Median for Deep Convolutional Neural Networks AccelerationCVPRFFPGMPyTorch(Author)Image Classification2019
02Compressing Convolutional Neural Networks via Factorized Convolutional FiltersCVPRFFCFPyTorch(Author)Image Classification2019
03Rigging the Lottery: Making All Tickets WinnersICMLWRigLPyTorch(Author)Image Classification2019
04NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune ParadigmarXivNNeST-Image Classification2019
05Variational Convolutional Neural Network PruningCVPRFVCP-Image Classification2019
06Sparse Networks from Scratch: Faster Training without Losing PerformancearXivWSMPyTorch(Author)Image Classification2019
07Online Filter Clustering and Pruning for Efficient ConvetsarXivW--Image Classification2019
08Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse ReparameterizationICMLWDSRPyTorch(Not Available)Image Classification2019
09Network Pruning via Transformable Architecture SearchNeurIPSFTASPyTorch(Author)Image Classification2019
10MetaPruning: Meta Learning for Automatic Neural Network Channel PruningICCVFMetaPruningPyTorch(Author)Image Classification2019
11DHP: Differentiable Meta Pruning via HyperNetworksECCVFDHPPyTorch(Author)Image Classification&Super-resolution&Denoising2019
12Global Sparse Momentum SGD for Pruning Very Deep Neural NetworksNeurIPSWGSMPyTorch(Author)Image Classification2019
Pruning During Training CNNs 2018 and earlier
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Learning the Number of Neurons in Deep NetworksNIPSN--Image Classification2016
02Learning Structured Sparsity in Deep Neural NetworksNIPSFCSSLCaffe(Author)Image Classification2016
03Learning Efficient Convolutional Networks through Networks SlimmingICCVCSlimmingLua(Author)Image Classification2017
04Deep Rewiring: Training very Sparse Deep NetworksICLRW--Image Classification&Audio2018
05Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution LayersICLRF-TensorFlow(Author) PyTorch(3rd)Image Classification&Segmentation2018
06Data-Driven Sparse Structure Selection for Deep Neural NetworksECCVFSSSMXNet(Author)Image Classification2018
07MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep NetworksCVPRLMorphNetPyTorch(Author)Image Classification2018
08Learning Sparse Neural Networks Through $L_0$ RegularizationICLRFN-PyTorch(Author)Image Classification2018
09Soft Filter Pruning for Accelerating Deep Convolutional Neural NetworksIJCAIFSFPPyTorch(Author)Image Classification2018
10Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network ScienceNature CommunicationW&PSET-Image Classification2018
1.1.2.2 Pruning Other Models
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Exploring Sparsity in Recurrent Neural NetworksICLRW-PyTorchSpeech Recognition2017
02Sparse Training via Boosting Pruning Plasticity with NeuroregenerationNeurIPSHGraNetPyTorchImage Classification2021
03Selfish Sparse RNN TrainingICMLWSNT-ASGDPyTorch(Anthor)Language Modeling2021
04Dynamic Sparse Training for Deep Reinforcement LearningIJCAIW-PyTorch(Anthor)Continuous Control2022
05The State of Sparse Training in Deep Reinforcement Learning.ICMLW-Tensorflow(Anthor)Continuous Control2022

1.1.3 Pruning After Training

1.1.3.1 Pruning CNNs
Pruning After Training CNNs 2024
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Fast and Controllable Post-training Sparsity: Learning Optimal Sparsity Allocation with Global Constraint in MinutesAAAIWFCPTS-Image Classification&Object Detection2024
02UPDP: A Unified Progressive Depth Pruner for CNN and Vision TransformerAAAILUPDP-Image Classification&Object Detection2024
Pruning After Training CNNs 2023
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Unified Data-Free Compression: Pruning and Quantization without Fine-TuningICCVCUDFC-Image Classification2023
02Unmasking the Lottery Ticket Hypothesis: What’s Encoded in a Winning Ticket’s Mask?ICLR(TOP-25%)W--Image Classification2023
03DepGraph: Towards Any Structural PruningCVPRCDepGraphPyTorch(Author)CV/NLP2023
04DFPC: Data flow driven pruning of coupled channels without dataICLRCDFPCPyTorch(Author)Image Classification2023
05Memory-Oriented Structural Pruning for Efficient Image RestorationAAAICMOSP-Image Restoration2023
06Trainability Preserving Nueral Structured PruningICLRFTPPPytorch(Author)Image Classification2023
Pruning After Training CNNs 2022
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Gradient Flow in Sparse Neural Networks and How Lottery Tickets WinAAAIW-PyTorch(Author)Image Classification2022
02How Well Do Sparse ImageNet Models Transfer?CVPRW-PyTorch(Author)Image Classification&Object Detection2022
03Lottery Jackpots Exist in Pre-trained ModelsarXivWoBERTPyTorch(Author)Image Classification2022
04Graph Pruning for Model CompressionApplied IntelligenceWGraphPruning-Image Classification2022
05Advancing Model Pruning via Bi-level OptimizationNeurIPSWCBiPPyTorch(Author)Image Classification2022
06Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and PruningNeurIPSWExactOBSPyTorch(Author)Image Classification&Object Detection&Question Answering2022
07Prune Your Model Before Distill ItECCVF-PyTorch(Author)Image Classification2022
08SOSP: Efficiently Capturing Global Correlations by Second-Order Structured PruningICLR (Spotlight)FSOSPPyTorch(Author)(Releasing)Image Classification2022
09Dreaming to Prune Image Deraining NetworksTPAMI1XN-PyTorch(Author)Image Classification2022
101xN Pattern for Pruning Convolutional Neural NetworksCVPRF--Image Deraining2022
11Prior Gradient Mask Guided Pruning-Aware Fine-TuningAAAICPGMPFPyTorch(Author)Image Classification2022
Pruning After Training CNNs 2021
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01ChipNet: Budget-Aware Pruning with Heaviside Continuous ApproximationsICLRFChipNetPyTorch(Author)Image Classification2021
02Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?NeurIPSW-PyTorch(Author)Image Classification2021
03Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted NetworkICLRWMPTsPyTorch(Author)Image Classification2021
04Long live the lottery: the existence of winning tickets in lifelong learningICLRW-PyTorch(Author)Image Classification2021
05Enabling Retrain-free Deep Neural Network Pruning Using Surrogate Lagrangian RelaxationIJCAIW--Image Classification & Object Detection2021
06Joint-DetNAS: Upgrade Your Detector with NAS, Pruning and Dynamic DistillationCVPRFJoint-DetNAS-Image Classification & Object Detection2021
07Validating the Lottery Ticket Hypothesis with Inertial Manifold TheoryNeurIPSW--Image Classification2021
08Towards Compact CNNs via Collaborative CompressionCVPRFCCPyTorch(Author)Image Classification2021
09NPAS: A Compiler-aware Framework of Unified Network Pruning andArchitecture Search for Beyond Real-Time Mobile AccelerationCVPRFNPAS-Image Classification2021
10Neural Pruning via Growing RegularizationICLRWFGreg-Image Classification2021
11Towards Adversarial Robustness Via Compact Feature RepresentationsICASSPN-PyTorch(Author)Adversarial Robustness2021
12On the Predictability of Pruning Across ScalesICMLW--Image Classification2021
13How much pre-training is enough to discover a good subnetwork?arXivW--Image Classification2021
14The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision ModelsCVPRW-PyTorch(Author)Image Classification2021
15The Elastic Lottery Ticket HypothesisNeurIPSWE-LTHPyTorch(Author)Image Classification2021
16Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable MasksNeurIPSN:MAdaPrunePyTorch(Author)Image Classification2021
17Why Lottery Ticket Wins? A Theoretical Perspective of Sample Complexity on Sparse Neural NetworksNeurIPSW--Image Classification2021
18Group Fisher Pruning for Practical Network CompressionICMLFGFPPyTorch(Author)Image Classification&Object Detection2021
19TransTailor: Pruning the Pre-trained Model for Improved Transfer LearningAAAIFTransTailor-Image Classification2021
20Network Pruning That Matters: A Case Study on Retraining VariantsICLRF-PyTorch(Author)Image Classification2021
21The Lottery Ticket Hypothesis for Object RecognitionCVPRW-PyTorch(Author)Object Detection2021
22Lottery Jackpot Exist in Pre-trained ModelsTPAMIWJackpotPyTorch(Author)Image Classification2021
23Accelerate CNNs from Three Dimensions: A Comprehensive Pruning FrameworkICMLF--Image Classification2021
24Network Pruning via Performance MaximizationCVPRFNPPMPytorch(Author)Image Classification2021
25Accelerating Sparse Deep Neural NetworksarXivW--Image Classification&Image Segmentation and Detection&Language Modeling&Language Translation2021
Pruning After Training CNNs 2020
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01SCOP: Scientific Control for Reliable Neural Network PruningNeurIPSFSCOPPyTorch(Author)Image Classification2020
02Discrete Model Compression With Resource Constraint for Deep Neural NetworksCVPRF--Image Classification2020
03HRank: Filter Pruning using High-Rank Feature MapCVPRFHRankPytorch(Author)Image Classification2020
04Learning Filter Pruning Criteria for Deep Convolutional Neural Networks AccelerationCVPRFLFPC-Image Classification2020
05Towards Efficient Model Compression via Learned Global RankingCVPRFLeGRPytorch(Author)Image Classification2020
06Reborn filters: Pruning convolutional neural networks with limited dataAAAIF--Image Classification2020
07Operation-Aware Soft Channel Pruning using Differentiable MasksICMLFSCP-Image Classification2020
08Neural Network Pruning with Residual-Connections and Limited-DataCVPRCCURLPyTorch(Author)Image Classification2020
09On the Transferability of Winning Tickets in Non-Natural Image DatasetsarXivW--Image Classification2020
10Towards Compact and Robust Deep NetworksarXivW--Image Classification2020
11HYDRA: Pruning Adversarially Robust Neural NetworksNeurIPSWHYDRAPyTorch(Author)Adversarial Robustness2020
12Movement Pruning: Adaptive Sparsity by Fine-TuningNeurIPSW-PyTorch(Author)NLP2020
13DMCP: Differentiable Markov Channel Pruning for Neural NetworksCVPRCDMCP-Image Classification2020
14How many winning tickets are there in one DNN?arXivW--Image Classification2020
15Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network CompressionCVPRFHingePyTorch(Author)Image Classification2020
16Proving the Lottery Ticket Hypothesis for Convolutional Neural NetworksICMLN---2020
17Logarithmic Pruning is All You NeedNeurIPSN---2020
18Optimal Lottery Tickets via SUBSETSUM:Logarithmic Over-Parameterization is SufficientNeurIPSN-PyTorch(Author)Image Classification2020
19EagleEye: Fast Sub-net Evaluation for Efficient Neural Network PruningECCVFEagleEyePyTorch(Author)Image Classification2020
20Channel Pruning via Automatic Structure SearchIJCAIFABCPyTorch(Author)Image Classification2020
Pruning After Training CNNs 2019
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Auto-Balanced Filter Pruning for Efficient Convolutional Neural NetworksAAAIF--Image Classification2019
02Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural NetworksNeurIPSFGate DecoratorPyTorch(Author)Image Classification&Semantic Segmentation2019
03EigenDamage: Structured Pruning in the Kronecker-Factored EigenbasisICMLCEigenDamagePyTorch(Author)Image Classification2019
04Importance Estimation for Neural Network PruningCVPRFTaylor-FO-BNPyTorch(Author)Image Classification2019
05The State of Sparsity in Deep Neural NetworksarXivw-TensorFlow(Author)Image Classification&machine translation2019
06Collaborative Channel Pruning for Deep NetworksICMLFCCP-Image Classification2019
07One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizersNeurIPSW--Image Classification2019
08ECC: Platform-Independent Energy-Constrained Deep Neural Network Compression via a Bilinear Regression ModelCVPRFECCPytorch(Author)Image Classification&Semantic Segmentation2019
09Approximated Oracle Filter Pruning for Destructive CNN Width Optimization githubICMLFAOFPPytorch(Author)Image Classification2019
10Sparse Transfer Learning via Winning Lottery TicketsarXivW-PyTorch(Author)Image Classification2019
11Global Sparse Momentum SGD for Pruning Very Deep Neural NetworksNeurIPSW-PyTorch(Author)Image Classification2019
12The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural NetworksICLR (Best)WLTHTensorFlow(Author)Image Classification2019
13Deconstructing Lottery Tickets: Zeros, Signs, and the SupermaskNeurIPSW-TensorFlow(Author)Image Classification2019
14Winning the Lottery with Continuous SparsificationNeurIPSFCSPyTorch(Author)Image Classification2019
15Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated StructureCVPRFC-SGDTensorflow(Author)Image Classification2019
16Exploiting Kernel Sparsity and Entropy for Interpretable CNN CompressionCVPRWKSEPyTorch(Author)Image Classification2019
17Towards Compact ConvNets via Structure-Sparsity Regularized Filter PruningTNNLSFSSRCaffe(Author)Image Classification2019
18Towards Optimal Structured CNN Pruning via Generative Adversarial LearningCVPRFGALPyTorch(Author)Image Classification2019
18Efficient Neural Network CompressionCVPRCENCPyTorch(Author)Image Classification2019
Pruning After Training CNNs 2018
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Accelerating Convolutional Networks via Global & Dynamic Filter PruningIJCAIFGDP-Image Classification2018
02AMC: Automl for model compression and acceleration on mobile devicesECCVFAMCTensorFlow(3rd)Image Classification2018
03Exploring Linear Relationship in Feature Map Subspace for ConvNets CompressionarXivF--Object Detection&Human Pose Estimation2018
04To prune, or not to prune: exploring the efficacy of pruning for model compressionICLRWW-TensorFlow(Author)NLP2018
05CLIP-Q: Deep Network Compression Learning by In-Parallel Pruning-QuantizationCVPRWCLIP-Q-Image Classification2018
06Discrimination-aware Channel Pruning for Deep Neural NetworksNeurIPSCDCPTensorFlow(Author)Image Classification2018
07NISP: Pruning Networks using Neuron Importance Score PropagationCVPRNCNISP-Image Classification2018
082PFPCE: Two-Phase Filter Pruning Based on Conditional EntropyAAAIW2PFPCE-Image Classification2018
Pruning After Training CNNs 2017 and earlier
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Optimal Brain DamageNIPSWOBD-Image Classification1989
02Second Order Derivatives for Network Pruning: Optimal Brain SurgeonNIPSWOBS-Image Classification1992
03Structured Pruning of Deep Convolutional Neural NetworksarXivC--Image Classification2015
04Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman CodingICLR (Best)W-Caffe(Author)Image Classification2016
05ThiNet: A Filter Level Pruning Method for Deep Neural Network CompressionICCV&TPAMIFThiNetCaffe(Author), PyTorch(3rd)Image Classification2017&2019
06Pruning Convolutional Neural Networks for Resource Efficient InferenceICLRF-PyTorchImage Classification2017
07Pruning Filters for Efficient ConvNetsICLRFPFECPyTorch(3rd)Image Classification2017
08Channel pruning for accelerating very deep neural networksICCVC-Caffe(Author)Image Classification&Object Detection2017
1.1.3.2 Pruning ViTs
Pruning After Training ViTs 2024
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Fast and Controllable Post-training Sparsity: Learning Optimal Sparsity Allocation with Global Constraint in MinutesAAAIWFCPTS-Image Classification&Object Detection2024
02UPDP: A Unified Progressive Depth Pruner for CNN and Vision TransformerAAAILUPDP-Image Classification&Object Detection2024
03Pruning Self-attentions into Convolutional Layers in Single PathTPAMIHSPViTPyTorchImage Classification&Object Detection2024
Pruning After Training ViTs 2023
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01X-Pruner: eXplainable Pruning for Vision TransformersCVPRCHX-PrunerPytorch(Author)Image Classification2023
02Global Vision Transformer Pruning with Hessian-Aware SaliencyCVPRCHNViT-Image Classification2023
03Pruning Parameterization with Bi-level Optimization for Efficient Semantic Segmentation on the EdgeCVPRWSTE-semantic Segmentation2023
04Instant Soup: Cheap Pruning Ensembles in A Single Pass Can Draw Lottery Tickets from Large ModelsICMLWISPPytorch(Author)Image Classification&NLP2023
Pruning After Training ViTs 2022
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Width & Depth Pruning for Vision TransformersAAAICWDPruningPytorch(Author)Image Classification2022
02SAViT: Structure-Aware Vision Transformer Pruning via Collaborative OptimizationNeurIPSCHESAViTPytorch(Author)Image Classification&object detection2022
03VTC-LFC: Vision Transformer Compression with Low-Frequency ComponentsNeurIPSCVTC-LFCPytorch(Author)Image Classification2022
04CP-ViT: Cascade Vision Transformer Pruning via Progressive Sparsity PredictionarXivHCP-ViT-Image Classification2022
05Unified Visual Transformer CompressionICLRHUVCPytorch(Author)Image Classification2022
1.1.3.3 Pruning BERTs
Pruning After Training BERTs 2023
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse ApproximationICMLHLoSparsePyTorch(Author)NLP2023
02Instant Soup: Cheap Pruning Ensembles in A Single Pass Can Draw Lottery Tickets from Large ModelsICMLWISPPytorch(Author)Image Classification&NLP2023
03Gradient-Free Structured Pruning with Unlabeled DataICMLFKCM-NLP2023
04The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that MatterarXivW&N:M-Pytorch(Author)NLP2023
Pruning After Training BERTs 2022
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Structured Pruning Learns Compact and Accurate ModelsACLLHCoFiPyTorch(Author)Natural Language Understanding2022
02From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model CompressionAAAIWHCAPPyTorch(Author)NLP2022
03PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight ImportanceICMLWCPLATONPyTorch(Author)Natural Language Understanding&Question Answering&Image Classification2022
04Parameter-Efficient Sparsity for Large Language Models Fine-TuningIJCAIWPSTPyTorch(Author)Language Modeling2022
05The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language ModelsEMNLPWoBERTPyTorch(Author)Natural Language Understanding2022
06Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and PruningNeurIPSWExactOBSPyTorch(Author)Image Classification&Object Detection&Question Answering2022
Pruning After Training BERTs 2021
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Super Tickets in Pre-Trained Language Models: From Model Compression to Improving GeneralizationACLWsuper ticketsPyTorch(Author)Language Understanding2021
02Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable MasksNeurIPSN:MAdaPrunePyTorch(Author)Image Classification2021
03Prune Once for All: Sparse Pre-Trained Language ModelsNeurIPS WorkshopWOFAPyTorch(Author)NLP2021
04BERT Busters: Outlier Dimensions that Disrupt TransformersACLW--NLP2021
05PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech RecognitionNeurIPSWPARP-Speach Recognition2021
06Parameter-Efficient Transfer Learning with Diff PruningACLMDiff PruningPyTorch(Author)NLP2021
07EarlyBERT: Efficient BERT training via early-bird lottery ticketsACL-IJCNLPHEarlyBERTPyTorch(Author)NLP2021
08The Lottery Ticket Hypothesis for Pre-trained BERT NetworksICMLW-PyTorch(Author)Language Modeling2021
09Structured Pruning of Large Language ModelsarXivWFLOPPyTorch(Author)NLP classification2021
10Accelerating Sparse Deep Neural NetworksarXivW--Image Classification&Image Segmentation and Detection&Language Modeling&Language Translation2021
11Differentiable Subset Pruning of Transformer HeadsTACLH-PyTorch(Author)NLP2021
Pruning After Training BERTs 2020
No.TitleVenueTypeAlgorithm NameCodeAPPYear
03Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of TransformersICMLW--NLP2020
04When BERT Plays the Lottery, All Tickets Are WinningEMNLPW-PyTorch(Author)Language Modeling2020
05LadaBERT: Lightweight Adaptation of BERT through Hybrid Model CompressionCOLINGW--NLP(Sentiment Classification,Natural Language Inference,Pairwise Semantic Equivalence)2020
06Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity PriorEMNLPH--NLP2020
07Compressing BERT: Studying the Effects of Weight Pruning on Transfer LearningRep4NLPW--NLP2020
Pruning After Training BERTs 2019
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Reweighted Proximal Pruning for Large-Scale Language RepresentationarXivOther--NLP2019
02Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured PruningEMNLPOther--NLP2019
1.1.3.4 Pruning LLMs
Pruning After Training LLMs 2024
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-TuningACLCHLoRAPrunePyTorch(Author)Language Modeling&Classification2024
02A Simple and Effective Pruning Approach for Large Language ModelsICLRWWandaPyTorch(Author)Language Modeling&Classification2024
03SliceGPT: Compress Large Language Models by Deleting Rows and ColumnsICLRCHSliceGPTPyTorch(Author)Language Modeling&Classification2024
04Fluctuation-based Adaptive Structured Pruning for Large Language ModelsAAAICHFLAPPyTorch(Author)Language Modeling&Classification2024
05BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity AllocationarXivBBESAPyTorch(Author)Language Modeling&Classification2024
06APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and InferenceICMLHCAPTPyTorch(Author)Classification2024
07Sheared LLaMA: Accelerating Language Model Pre-training via Structured PruningICLRCHSheared LLaMAPyTorch(Author)Language Modeling&Classification2024
08Everybody Prune Now: Structured Pruning of LLMs with only Forward PassesarXivCHBonsaiPyTorch(Author)Language Modeling&Classification2024
09LaCo: Large Language Model Pruning via Layer CollapsearXivLLaCo-Language Modeling&Classification2024
10ShortGPT: Layers in Large Language Models are More Redundant Than You ExpectarXivLShortGPT-Language Modeling&Classification2024
11SparseLLM: Towards Global Pruning for Pre-trained Language ModelsarXivBSparseLLMPyTorch(Author)Language Modeling&Classification2024
12SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer BlocksarXivNSLEBPyTorch(Author)Language Modeling&Classification2024
13Streamlining Redundant Layers to Compress Large Language ModelsarXivLLLMStreamline-Language Modeling&Classification2024
14Why Lift so Heavy? Slimming Large Language Models by Cutting Off the LayersarXivL--Classification2024
15Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining MethodsICLRWHC-PyTorch(Author)Classification2024
16Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured SparsityVLDBWFlash-LLMPyTorch(Author)Recognizing Textual Entailment2024
17The LLM SurgeonarXivWCLLM SurgeonPyTorch(Author)Language Modeling2024
18Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High SparsityICMLWOWLPyTorch(Author)Language Modeling&Classification2024
19The Unreasonable Ineffectiveness of the Deeper LayersarXivB--Classification2024
20Enhancing One-Shot Pruned Generative Pre-training Language Models through Sparse-Dense-Sparse MechanismOpenReviewWSDS-Classification2024
21KS-Lottery: Finding Certified Lottery Tickets for Multilingual Language ModelsarXivW--Language Translation2024
Pruning After Training LLMs 2023
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01SparseGPT: Massive Language Models Can be Accurately Pruned in One-ShotNeurIPSWP-PyTorch(Author)Language Modeling&Classification2023
02LLM-Pruner: On the Structural Pruning of Large Language ModelsarXivCHPLLM-PrunerPyTorch(Author)Language Modeling&Language Generation&Classification2023
03LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge RecoveryarXivCHLoRAShear-Language Modeling&Language Generation&Classification2023
04Compresso: Structured Pruning with Collaborative Prompting Learns Compact Large Language ModelsarXivCHCompressoPyTorch(Author)Classification2023
05Mini-GPTs: Efficient Large Language Models through Contextual PruningarXivWC--Language Modeling& Classification2023
06The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that MatterarXivW&N:M-Pytorch(Author)NLP2023
1.1.3.5 Pruning Diffusion Models
Pruning After Training Diffusion Models 2023
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Structural Pruning for Diffusion ModelsNeurIPSCDiff-PruningPyTorch(Author)Image Generation2023
1.1.3.6 Pruning Vision-and-Languages
Pruning After Training VLMs 2024
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language ModelsICLRLECoFLaPPytorch(Author)VQA&Image Captioning&Image-text Retrieval&Image Classification2024
Pruning After Training VLMs 2023
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Large Multimodal Model Compression via Efficient Pruning and Distillation at AntGrouparXivB--Multimodal Advertisement Audition2023
02UPop: Unified and Progressive Pruning for Compressing Vision-Language TransformersICMLHUPopPytorch(Author)Image Classification&Image Caption&Image Retrieval&VQA2023
03Instant Soup: Cheap Pruning Ensembles in A Single Pass Can Draw Lottery Tickets from Large ModelsICMLWISPPytorch(Author)Image Classification&NLP2023
Pruning After Training VLMs 2022
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Playing Lottery Tickets with Vision and LanguageAAAIW--Vision-and-Language2022
1.1.3.7 Pruning Other Models
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be PrunedACLW-PyTorch(Author)NLP2019
02Playing the Lottery with Rewards and Multiple Languages: Lottery Tickets in RL and NLPICLRW--Classic Control&Atari Game2020
03Dynamic Sparsity Neural Networks for Automatic Speech RecognitionICASSPW--Speach Recognition2021
04GAN Compression: Efficient Architectures for Interactive Conditional GANsarXivC--Image-to-Image Translation2021
05Content-Aware GAN CompressionCVPRF-PyTorch(Author)Image Generation, Image Projection, Image Editing
06A Unified Lottery Ticket Hypothesis for Graph Neural NetworksICMLW-PyTorch(Author)Node Classification&Link Prediction2021
07Winning Lottery Tickets in Deep Generative ModelsAAAIW--Image generative2021
08GANs Can Play Lottery Tickets TooICLRW-PyTorch(Author)Image generative2021
09Layer-wise Pruning of Transformer Attention Heads for Efficient Language ModelingarXivH-PyTorch(Author)Lanugage Modeling2021
10Can We Find Strong Lottery Tickets in Generative Models?arXivW--Image generative2022
11Exploring Lottery Ticket Hypothesis in Spiking Neural NetworksECCVWETPyTorch(Author)Image Classification2022
12Structured Pruning for Efficient Generative Pre-trained Language ModelsACLCCP3-Language Modeling&Machine Translation&Abstractive Summarization2023
13Rethinking Graph Lottery Tickets: Graph Sparsity MattersICLRW--Node Classification2023
14CP3: Channel Pruning Plug-in for Point-based NetworksCVPRCCP3-3D Image Classification and Object Detection2023
1.1.3.8 Post Training
Post Training 2024
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Fast and Controllable Post-training Sparsity: Learning Optimal Sparsity Allocation with Global Constraint in MinutesAAAIWFCPTS-Image Classification2024
Post Training 2023
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01SparseGPT: Massive Language Models Can be Accurately Pruned in One-ShotNeurIPSWP-PyTorch(Author)Language Modeling2023
02Unified Data-Free Compression: Pruning and Quantization without Fine-TuningICCVCUDFC-Image Classification2023
03OTOv3: Automatic Architecture-Agnostic Neural Network Training and Compression from Structured Pruning to Erasing OperatorsarXivWFC--Image Classification2023
Post Training 2022
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01CP-ViT: Cascade Vision Transformer Pruning via Progressive Sparsity PredictionarXivHCP-ViT-Image Classification2022
02Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and PruningNeurIPSWExactOBSPyTorch(Author)Image Classification&Object Detection&Question Answering2022
03A Fast Post-Training Pruning Framework for TransformersNeurIPSHF-PyTorch(Author)Natural Language Understanding2022
Post Training 2021
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Enabling Retrain-free Deep Neural Network Pruning Using Surrogate Lagrangian RelaxationIJCAIW--Image Classification & Object Detection2021
02Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable MasksNeurIPSN:MAdaPrunePyTorch(Author)Image Classification2021

1.1.4 Pruning in Early Training

No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Linear Mode Connectivity and the Lottery Ticket HypothesisICMLW--Image Classification2020
02When To Prune? A Policy Towards Early Structural PruningCVPRFPaT-Image Classification2022
03Drawing Early-Bird Tickets: Towards More Efficient Training of Deep NetworksICLRW-PyTorch(Author)Image Classification2020
04A Gradient Flow Framework For Analyzing Network PruningICLRF-PyTorch(Author)Image Classification2021

1.2 Dynamic Pruning

No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Channel Gating Neural NetworksNeurIPSFRNP-Image Classification2017
02Channel Gating Neural NetworksNeurIPSCCGNetPyTorch(Author)Image Classification2019
03Dynamic Channel Pruning: Feature Boosting and SuppressionICLRCFBSPyTorch(Author)Image Classification2019
04Frequency-Domain Dynamic Pruning for Convolutional Neural NetworksNeurIPSFFDNP-Image Classification2019
05Fire Together Wire Together: A Dynamic Pruning Approach With Self-Supervised Mask PredictionCVPRF--Image Classification2019
06Dynamic Dual Gating Neural NetworksICCVCDGNetPyTorch(Author)Image Classification2021
07Manifold Regularized Dynamic Network PruningCVPRFManiDPPyTorch(Author)Image Classification2021
08Contrastive Dual Gating: Learning Sparse Features With Contrastive LearningCVPRWFCDG-Image Classification2022

2. Learning and Pruning

2.1 Continual learning

No.TitleVenueAlgorithm NameCodeAPPYear
01Continual Learning via Neural PruningarXivCLNP-Image Classification2019
02Learning Bayesian Sparse Networks With Full Experience Replay for Continual LearningCVPRSNCL-Image Classification2022
03Continual Prune-and-Select: Class-Incremental Learning with SPecialized SubnetworksApplied Intelligence-PyTorch(Author)Image Classification2023
04Continual Domain Adaptation through Pruning-aided Domain-specific Weight ModulationCVPRWPaCDAPyTorch(Author)Image Classification2023

2.2 Contrastive learning

No.TitleVenueAlgorithm NameCodeAPPYear
01Studying the impact of magnitude pruning on contrastive learning methodsICML-PyTorch(Author)Image Classification2020
02Training Debiased Subnetworks with Contrastive Weight PruningCVPRDCWP-Image Classification2023

2.3 Federated learning

No.TitleVenueAlgorithm NameCodeAPPYear
01FedDUAP: Federated Learning with Dynamic Update and Adaptive Pruning Using Shared Data on the ServerIJCAIFedDUAP-Image Classification2020
02Model Pruning Enables Efficient Federated Learning on Edge DevicesTNNLS-PyTorch(Author)Image Classification2022

3. Application

3.1 Computer Vision

No.TitleVenueCodeAPPYear
01Deep Rewiring: Training very Sparse Deep NetworksICLR-Image Classification&Audio2018
02Co-Evolutionary Compression for Unpaired Image TranslationICCVPyTorch(Author)Image Style Translation2019
03Content-Aware GAN CompressionCVPRPyTorch(Author)Image Style Translation2021
04Training Neural Networks with Fixed Sparse MasksNeurIPSPyTorch(Author)Image Classification2021
05Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization SpaceCVPRPyTorch(Author)Image Classification&Audio2022
06SuperTickets: Drawing Task-Agnostic Lottery Tickets from Supernets via Jointly Architecture Searching and Parameter PruningECCVPyTorch(Author)Image Classification&Object Detection&Human Pose Estimation2022

3.2 Natural Language Processing

No.TitleVenueCodeAPPYear
01When BERT Plays the Lottery, All Tickets Are WinningEMNLPPyTorch(Author)Language Modeling2020
02The Lottery Ticket Hypothesis for Pre-trained BERT NetworksICMLPyTorch(Author)Language Modeling2021
03Structured Pruning Learns Compact and Accurate ModelsACLPyTorch(Author)Natural Language Understanding2022
04A Fast Post-Training Pruning Framework for TransformersNeurIPSPyTorch(Author)Natural Language Understanding2022
05A Fast Post-Training Pruning Framework for TransformersNeurIPSPyTorch(Author)Natural Language Understanding2022
06The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language ModelsEMNLPPyTorch(Author)Natural Language Understanding2022
07Pruning Meets Low-Rank Parameter-efficientarXiv-Image Classification&Language Modeling2023
08LLM-Pruner: On the Structural Pruning of Large Language ModelsarXiv-Language Modeling2023

3.3 Audio Signal Processing

No.TitleVenueCodeAPPYear
01Exploring Sparsity in recurrent neural networksICLRPyTorchSpeech Recognition2017
02Deep Rewiring: Training very Sparse Deep NetworksICLR-Image Classification&Audio2018

4. Combination

4.1 Pruning and Quantization

No.TitleVenueCodeAPPYear
01CLIP-Q: Deep Network Compression Learning by In-Parallel Pruning-QuantizationCVPR-Image Classification2018
02Accelerating Sparse Deep Neural NetworksarXiv-Image Classification&Object Detection&Language Translation&Language Modeling&Image Synthesis&Domain Translation&Style Transfer&Image-Image Translation&Super Resolution2021
03OPQ: Compressing Deep Neural Networks with One-shot Pruning-QuantizationAAAI-Image Classification2021
04Deep Model Compression Based on the Training HistoryarXiv-Image Classification2022
05LLM-Pruner: On the Structural Pruning of Large Language ModelsarXivPyTorchCausal Language Modeling2023
06Unified Data-Free Compression: Pruning and Quantization without Fine-TuningICCV-Image Classification2023

5. Survey of Pruning

Survey of Pruning 2024

No.TitleVenueCodeAPPYear
01Structured Pruning for Deep Convolutional Neural Networks: A surveyTPAMI-CV&NLP2024
02A survey on efficient vision transformers: algorithms, techniques, and performance benchmarkingarXiv-CV2024
03A Survey of Lottery Ticket HypothesisarXiv-CV&NLP2024
04Model Compression and Efficient Inference for Large Language Models: A SurveyarXiv-NLP2024

Survey of Pruning 2023

No.TitleVenueCodeAPPYear
01Why is the State of Neural Network Pruning so Confusing? On the Fairness, Comparison Setup, and Trainability in Network PruningarXivPyTorch(Author)Image Classification2023
02Transforming Large-Size to Lightweight Deep Neural Networks for IoT ApplicationsACM Computing Surveys-CV&NLP&Audio2023
03A Survey on Model Compression for Large Language ModelsTACL-NLP&Unseen Instructions2023
04Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to SystemsarXiv--2023
05A Survey on Dynamic Neural Networks for Natural Language ProcessingarXiv-NLP2023
06Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a SurveyarXiv-CV&NLP2023

Survey of Pruning 2022

No.TitleVenueCodeAPPYear
01A Survey on Efficient Convolutional Neural Networks and Hardware AccelerationElectronics--2022
02Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a SurveyarXiv-Image Classification2022
03Efficient Transformers: A SurveyarXiv-CV&NLP2022
04Recent Advances on Neural Network Pruning at InitializationIJCAI-CV&NLP2022

Survey of Pruning 2021

No.TitleVenueCodeAPPYear
01Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networksJMLR-Image Classification2021
02Dynamic Neural Networks: A SurveyarXiv--2021
03Pruning and Quantization for Deep Neural Network Acceleration: A SurveyNeurocomputing-Image Classification2021
04Compressing Large-Scale Transformer-Based Models: A Case Study on BERTTACL-NLP2021

Survey of Pruning 2020

No.TitleVenueCodeAPPYear
01Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive SurveyIEEE--2020
02Pruning Algorithms to Accelerate Convolutional Neural Networks for Edge Applications: A SurveyarXiv-Image Classification2020
03A Survey of Model Compression and Acceleration for Deep Neural NetworksarXiv--2020
04An Survey of Neural Network CompressionarXiv--2020
05Convolutional Neural Network Pruning: A SurveyCCC--2020
06What is the State of Neural Network Pruning?MLSys--2020
07A comprehensive survey on model compression and accelerationArtificial Intelligence Review--2020
08A Survey on Deep Neural Network Compression: Challenges, Overview, and SolutionsarXiv--2020

Survey of Pruning 2019 and earlier

No.TitleVenueCodeAPPYear
01Pruning Algorithms-A SurveyIEEE Transactions on Neural Networks-Image Classification1993
02Efficient Processing of Deep Neural Networks: A Tutorial and SurveyarXiv-Image Classification2017
03Recent advances in efficient computation of deep convolutional neural networksarXiv--2018
04The State of Sparsity in Deep Neural NetworksarXivPyTorch(Author)Image Classification&machine translation2019

6. Other Works

Papers

No.TitleVenueAlgorithm NameCodeAPPYear
01Is Pruning Compression?: Investigating Pruning Via Network Layer SimilarityWACV--Image Classification2020
02A Gradient Flow Framework For Analyzing Network PruningICLR-PyTorch(Author)Image Classification2021
03Data Level Lottery Ticket Hypothesis for Vision TransformersIJCAI-PyTorch(Author)Image Classification2021
04Are All Layers Created Equal?JMLR--Image Classification2022

https://github.com/airaria/TextPruner

Acknowledgements

We would like to express our gratitude to the authors of the articles cited in our survey and the authors of the following repositories.

https://github.com/he-y/awesome-Pruning/
https://github.com/MingSun-Tse/Awesome-Pruning-at-Initialization
https://github.com/csyhhu/Awesome-Deep-Neural-Network-Compression/blob/master/Paper/Pruning.md

Citation

If you find this project useful, please cite

@article{cheng2023survey,
  title={A Survey on Deep Neural Network Pruning:Taxonomy, Comparison, Analysis, and Recommendations},
  author={Hongrong Cheng and Miao Zhang and Javen Qinfeng Shi},
  journal={arXiv preprint arXiv:2308.06767},
  year={2023}
}

Contributors

hrcheng1066

340 commits

Similar repos

ghimiredhikura/Awasome-Pruning

Awesome Pruning. âś… Curated Resources for Neural Network Pruning.

178

MingSun-Tse/Efficient-Deep-Learning

Collection of recent methods on (deep) neural network compression and acceleration.

958

liyunqianggyn/Awesome-LLMs-Pruning

Awesome LLM pruning papers all-in-one repository with integrating all useful resources and…

180

VainF/Torch-Pruning

[CVPR 2023] DepGraph: Towards Any Structural Pruning; LLMs, Vision Foundation Models, etc.

3,357

cedrickchee/awesome-ml-model-compression

Awesome machine learning model compression research papers, quantization, tools, and learning…

546

horseee/Awesome-Efficient-LLM

A curated list for Efficient Large Language Models

2,043

Zhen-Dong/Awesome-Quantization-Papers

List of papers related to neural network quantization in recent AI conferences and journals.

853

windmaple/awesome-AutoML

Curating a list of AutoML-related research, tools, projects and other resources

944

hrcheng1066/awesome-pruning

309

340 commits

updated Aug 20, 2024

See the code

README

awesome-pruning

PRs WelcomeAwesome

Table of Contents

0. Overview

The repo includes the ongoing updates of representative neural network pruning papers and open-source codes.
Our paper [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations] (Paper Link), accepted by TPAMI 2024.

Taxonomy: In our survey, we provide a comprehensive review of the state-of-the-art in deep neural network pruning, which we categorize along five orthogonal axes: Universal/Specific Speedup, When to Prune, Pruning Criteria, Learn to Prune, and Fusion of Pruning and Other Techniques.

1. When to Prune

Type Explanation

TypeLFCNHBMEWPOther
ExplanationLayer pruningFilter pruningChannel pruningNeuron pruningHead pruningBlock pruningMatrix pruningEmbedding pruningWeight pruningPioneer workother types

1.1 Static Pruning

1.1.1 Pruning Before Training

1.1.1.1 Pruning CNNs
Pruning Before Training CNNs 2024
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01No Free Prune: Information-Theoretic Barriers to Pruning at InitializationICMLW--Image Classification2024
Pruning Before Training CNNs 2023
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Data-Free Model Pruning at Initialization via ExpandersCVPRWWRRegPyTorch(Author)Image Classification2023
02Revisiting Pruning as Initialization through the Lens of Ramanujan GraphICLR (TOP 5%)W-PyTorch(Author)Image Classification2023
03Pruning at Initialization - A Sketching PerspectivearXivW--Image Classification2023
04NTK-SAP: Improving neural network pruning by aligning training dynamicsICLRWNTK-SAPPyTorch(Author)Image Classification2023
Pruning Before Training CNNs 2022
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Prospect Pruning: Finding Trainable Weights at Initialization using Meta-GradientsICLRWFProsPrPyTorch(Author)Image Classification2022
02Dual Lottery Ticket HypothesisICLRWRSTPyTorch(Author)Image Classification2022
03Recent Advances on Neural Network Pruning at InitializationIJCAIW-PyTorch(Author)Image Classification2022
04The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse TrainingICLRW-PyTorch(Author)Image Classification2022
05Structured Pruning is All You Need for Pruning CNNs at InitializationarXivCPreCropping-Image Classification2022
Pruning Before Training CNNs 2021
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Progressive Skeletonization: Trimming More Fat from a network at initializationICLRWFORCEPyTorch(Author)Image Classification2021
02Robust Pruning at InitializationICLRWSPB-Image Classification2021
03A Unified Paths Perspective for Pruning at InitializationarXivW--Image Classification2021
04Prunining Neural Networks at Initialization: Why are We Missing the Mark?ICLRW--Image Classification2021
05Why is Pruning at Initialization Immune to Reinitializating and Shuffling?)arXivW--Image Classification2021
06Dense for the Price of Sparse: Improved Performance of Sparsely Initialized Networks via a Subspace Offset)ICMLWDCTpSPyTorch(Author)Image Classification2021
Pruning Before Training CNNs 2020
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Single Shot Structured Pruning Before TrainingarXivC3SPPyTorch(Author)Image Classification2020
02A Signal Propagation Perspective for Pruning Neural Networks at InitializationICLR (Spotlight)W-TensorFLow(Author)Image Classification2020
03Picking Winning Tickets before Training by Preserving Gradient Flow)ICLRWGraSPPyTorch(Author)Image Classification2020
04Pruning from ScratchAAAIC-PyTorch(Author)Image Classification2020
05Pruning neural networks without any data by iteratively conserving synaptic flowNeurIPSWSynFlowPyTorch(Author)Image Classification2020
06Sanity-Checking Pruning Methods: Random Tickets can Win the JackpotNeurIPSWSmart-RatiosPyTorch(Author)Image Classification2020
07Prunining via Iterative Ranking of Sensitivity StaticsarXivWFCSNIP-itPyTorch(Author)Image Classification2020
08What’s Hidden in a Randomly Weighted Neural Network?CVPRW-PyTorch(Author)Image Classification2020
09Finding trainable sparse networks through Neural Tangent TransferICMLW-PyTorch(Author)Image Classification2020
Pruning Before Training CNNs 2019
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01SNIP: Single-shot Network Pruning based on Connection SensitivityICLRWPSNIPTensorFLow(Author)Image Classification2019

1.1.2 Pruning During Training

1.1.2.1 Pruning CNNs
Pruning During Training CNNs 2024
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Auto-Train-Once: Controller Network Guided Automatic Network Pruning from ScratchCVPRWATOPyTorch(Author)Image Classification2024
Pruning During Training CNNs 2023
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01PDP: Parameter-free Differentiable Pruning is All You NeedNeurIPSWC--Vision&NLP2023
02LAPP: Layer Adaptive Progressive Pruning for Compressing CNNs from ScratcharXivFLAPP-Image Classification2023
Pruning During Training CNNs 2022
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01SuperTickets: Drawing Task-Agnostic Lottery Tickets from Supernets via Jointly Architecture Searching and Parameter PruningECCVWSuperTicketsPyTorch(Author)Image Classification&Object Detection&Human Pose Estimation2022
02Deep ensembling with no overhead for either training or testing: The all-round blessings of dynamic sparsityICLRWFreeTicketsPyTorch(Anthor)Image Classification2022
03Gradient Flow in Sparse Neural Networks and How Lottery Tickets WinAAAIW-PyTorch(Anthor)Image Classification2022
Pruning During Training CNNs 2021
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Training Adversarially Robust Sparse Networks via Bayesian Connectivity SamplingICMLW-PyTorch(Anthor)Adversarial Robustness2021
02Training Neural Networks with Fixed Sparse MasksNeurIPSW-PyTorch(Author)Image Classification2021
03DPFPS: Dynamic and Progressive Filter Pruning for Compressing Convolutional Neural Networks from ScratchAAAICDPFPSPyTorch(Author)Image Classification2021
04Sparse Training via Boosting Pruning Plasticity with NeuroregenerationNeurIPSWFGraNetPyTorch(Author)Image Classification2021
05Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse TrainingICMLWITOPPyTorch(Anthor)Image Classification2021
06Dense for the Price of Sparse: Improved Performance of Sparsely Initialized Networks via a Subspace OffsetICMLWDCTpSPyTorch(Anthor)Image Classification2021
Pruning During Training CNNs 2020
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Pruning Filter in FilterNeurIPSOtherSWPPyTorch(Author)Image Classification2020
02Dynamic Sparse Training: Find Effective Sparse Network from Scratch with Trainable Masked LayersICLRNFDSTPyTorch(Author)Image Classification2020
03DSA: More Efficient Budgeted Pruning via Differentiable Sparsity AllocationECCVFDSAPyTorch(Author)Image Classification2020
04Dynamic Model Pruning with FeedbackICLRWFDPFPyTorch(3rd)Image Classification2020
Pruning During Training CNNs 2019
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Filter Pruning via Geometric Median for Deep Convolutional Neural Networks AccelerationCVPRFFPGMPyTorch(Author)Image Classification2019
02Compressing Convolutional Neural Networks via Factorized Convolutional FiltersCVPRFFCFPyTorch(Author)Image Classification2019
03Rigging the Lottery: Making All Tickets WinnersICMLWRigLPyTorch(Author)Image Classification2019
04NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune ParadigmarXivNNeST-Image Classification2019
05Variational Convolutional Neural Network PruningCVPRFVCP-Image Classification2019
06Sparse Networks from Scratch: Faster Training without Losing PerformancearXivWSMPyTorch(Author)Image Classification2019
07Online Filter Clustering and Pruning for Efficient ConvetsarXivW--Image Classification2019
08Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse ReparameterizationICMLWDSRPyTorch(Not Available)Image Classification2019
09Network Pruning via Transformable Architecture SearchNeurIPSFTASPyTorch(Author)Image Classification2019
10MetaPruning: Meta Learning for Automatic Neural Network Channel PruningICCVFMetaPruningPyTorch(Author)Image Classification2019
11DHP: Differentiable Meta Pruning via HyperNetworksECCVFDHPPyTorch(Author)Image Classification&Super-resolution&Denoising2019
12Global Sparse Momentum SGD for Pruning Very Deep Neural NetworksNeurIPSWGSMPyTorch(Author)Image Classification2019
Pruning During Training CNNs 2018 and earlier
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Learning the Number of Neurons in Deep NetworksNIPSN--Image Classification2016
02Learning Structured Sparsity in Deep Neural NetworksNIPSFCSSLCaffe(Author)Image Classification2016
03Learning Efficient Convolutional Networks through Networks SlimmingICCVCSlimmingLua(Author)Image Classification2017
04Deep Rewiring: Training very Sparse Deep NetworksICLRW--Image Classification&Audio2018
05Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution LayersICLRF-TensorFlow(Author) PyTorch(3rd)Image Classification&Segmentation2018
06Data-Driven Sparse Structure Selection for Deep Neural NetworksECCVFSSSMXNet(Author)Image Classification2018
07MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep NetworksCVPRLMorphNetPyTorch(Author)Image Classification2018
08Learning Sparse Neural Networks Through $L_0$ RegularizationICLRFN-PyTorch(Author)Image Classification2018
09Soft Filter Pruning for Accelerating Deep Convolutional Neural NetworksIJCAIFSFPPyTorch(Author)Image Classification2018
10Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network ScienceNature CommunicationW&PSET-Image Classification2018
1.1.2.2 Pruning Other Models
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Exploring Sparsity in Recurrent Neural NetworksICLRW-PyTorchSpeech Recognition2017
02Sparse Training via Boosting Pruning Plasticity with NeuroregenerationNeurIPSHGraNetPyTorchImage Classification2021
03Selfish Sparse RNN TrainingICMLWSNT-ASGDPyTorch(Anthor)Language Modeling2021
04Dynamic Sparse Training for Deep Reinforcement LearningIJCAIW-PyTorch(Anthor)Continuous Control2022
05The State of Sparse Training in Deep Reinforcement Learning.ICMLW-Tensorflow(Anthor)Continuous Control2022

1.1.3 Pruning After Training

1.1.3.1 Pruning CNNs
Pruning After Training CNNs 2024
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Fast and Controllable Post-training Sparsity: Learning Optimal Sparsity Allocation with Global Constraint in MinutesAAAIWFCPTS-Image Classification&Object Detection2024
02UPDP: A Unified Progressive Depth Pruner for CNN and Vision TransformerAAAILUPDP-Image Classification&Object Detection2024
Pruning After Training CNNs 2023
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Unified Data-Free Compression: Pruning and Quantization without Fine-TuningICCVCUDFC-Image Classification2023
02Unmasking the Lottery Ticket Hypothesis: What’s Encoded in a Winning Ticket’s Mask?ICLR(TOP-25%)W--Image Classification2023
03DepGraph: Towards Any Structural PruningCVPRCDepGraphPyTorch(Author)CV/NLP2023
04DFPC: Data flow driven pruning of coupled channels without dataICLRCDFPCPyTorch(Author)Image Classification2023
05Memory-Oriented Structural Pruning for Efficient Image RestorationAAAICMOSP-Image Restoration2023
06Trainability Preserving Nueral Structured PruningICLRFTPPPytorch(Author)Image Classification2023
Pruning After Training CNNs 2022
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Gradient Flow in Sparse Neural Networks and How Lottery Tickets WinAAAIW-PyTorch(Author)Image Classification2022
02How Well Do Sparse ImageNet Models Transfer?CVPRW-PyTorch(Author)Image Classification&Object Detection2022
03Lottery Jackpots Exist in Pre-trained ModelsarXivWoBERTPyTorch(Author)Image Classification2022
04Graph Pruning for Model CompressionApplied IntelligenceWGraphPruning-Image Classification2022
05Advancing Model Pruning via Bi-level OptimizationNeurIPSWCBiPPyTorch(Author)Image Classification2022
06Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and PruningNeurIPSWExactOBSPyTorch(Author)Image Classification&Object Detection&Question Answering2022
07Prune Your Model Before Distill ItECCVF-PyTorch(Author)Image Classification2022
08SOSP: Efficiently Capturing Global Correlations by Second-Order Structured PruningICLR (Spotlight)FSOSPPyTorch(Author)(Releasing)Image Classification2022
09Dreaming to Prune Image Deraining NetworksTPAMI1XN-PyTorch(Author)Image Classification2022
101xN Pattern for Pruning Convolutional Neural NetworksCVPRF--Image Deraining2022
11Prior Gradient Mask Guided Pruning-Aware Fine-TuningAAAICPGMPFPyTorch(Author)Image Classification2022
Pruning After Training CNNs 2021
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01ChipNet: Budget-Aware Pruning with Heaviside Continuous ApproximationsICLRFChipNetPyTorch(Author)Image Classification2021
02Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?NeurIPSW-PyTorch(Author)Image Classification2021
03Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted NetworkICLRWMPTsPyTorch(Author)Image Classification2021
04Long live the lottery: the existence of winning tickets in lifelong learningICLRW-PyTorch(Author)Image Classification2021
05Enabling Retrain-free Deep Neural Network Pruning Using Surrogate Lagrangian RelaxationIJCAIW--Image Classification & Object Detection2021
06Joint-DetNAS: Upgrade Your Detector with NAS, Pruning and Dynamic DistillationCVPRFJoint-DetNAS-Image Classification & Object Detection2021
07Validating the Lottery Ticket Hypothesis with Inertial Manifold TheoryNeurIPSW--Image Classification2021
08Towards Compact CNNs via Collaborative CompressionCVPRFCCPyTorch(Author)Image Classification2021
09NPAS: A Compiler-aware Framework of Unified Network Pruning andArchitecture Search for Beyond Real-Time Mobile AccelerationCVPRFNPAS-Image Classification2021
10Neural Pruning via Growing RegularizationICLRWFGreg-Image Classification2021
11Towards Adversarial Robustness Via Compact Feature RepresentationsICASSPN-PyTorch(Author)Adversarial Robustness2021
12On the Predictability of Pruning Across ScalesICMLW--Image Classification2021
13How much pre-training is enough to discover a good subnetwork?arXivW--Image Classification2021
14The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision ModelsCVPRW-PyTorch(Author)Image Classification2021
15The Elastic Lottery Ticket HypothesisNeurIPSWE-LTHPyTorch(Author)Image Classification2021
16Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable MasksNeurIPSN:MAdaPrunePyTorch(Author)Image Classification2021
17Why Lottery Ticket Wins? A Theoretical Perspective of Sample Complexity on Sparse Neural NetworksNeurIPSW--Image Classification2021
18Group Fisher Pruning for Practical Network CompressionICMLFGFPPyTorch(Author)Image Classification&Object Detection2021
19TransTailor: Pruning the Pre-trained Model for Improved Transfer LearningAAAIFTransTailor-Image Classification2021
20Network Pruning That Matters: A Case Study on Retraining VariantsICLRF-PyTorch(Author)Image Classification2021
21The Lottery Ticket Hypothesis for Object RecognitionCVPRW-PyTorch(Author)Object Detection2021
22Lottery Jackpot Exist in Pre-trained ModelsTPAMIWJackpotPyTorch(Author)Image Classification2021
23Accelerate CNNs from Three Dimensions: A Comprehensive Pruning FrameworkICMLF--Image Classification2021
24Network Pruning via Performance MaximizationCVPRFNPPMPytorch(Author)Image Classification2021
25Accelerating Sparse Deep Neural NetworksarXivW--Image Classification&Image Segmentation and Detection&Language Modeling&Language Translation2021
Pruning After Training CNNs 2020
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01SCOP: Scientific Control for Reliable Neural Network PruningNeurIPSFSCOPPyTorch(Author)Image Classification2020
02Discrete Model Compression With Resource Constraint for Deep Neural NetworksCVPRF--Image Classification2020
03HRank: Filter Pruning using High-Rank Feature MapCVPRFHRankPytorch(Author)Image Classification2020
04Learning Filter Pruning Criteria for Deep Convolutional Neural Networks AccelerationCVPRFLFPC-Image Classification2020
05Towards Efficient Model Compression via Learned Global RankingCVPRFLeGRPytorch(Author)Image Classification2020
06Reborn filters: Pruning convolutional neural networks with limited dataAAAIF--Image Classification2020
07Operation-Aware Soft Channel Pruning using Differentiable MasksICMLFSCP-Image Classification2020
08Neural Network Pruning with Residual-Connections and Limited-DataCVPRCCURLPyTorch(Author)Image Classification2020
09On the Transferability of Winning Tickets in Non-Natural Image DatasetsarXivW--Image Classification2020
10Towards Compact and Robust Deep NetworksarXivW--Image Classification2020
11HYDRA: Pruning Adversarially Robust Neural NetworksNeurIPSWHYDRAPyTorch(Author)Adversarial Robustness2020
12Movement Pruning: Adaptive Sparsity by Fine-TuningNeurIPSW-PyTorch(Author)NLP2020
13DMCP: Differentiable Markov Channel Pruning for Neural NetworksCVPRCDMCP-Image Classification2020
14How many winning tickets are there in one DNN?arXivW--Image Classification2020
15Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network CompressionCVPRFHingePyTorch(Author)Image Classification2020
16Proving the Lottery Ticket Hypothesis for Convolutional Neural NetworksICMLN---2020
17Logarithmic Pruning is All You NeedNeurIPSN---2020
18Optimal Lottery Tickets via SUBSETSUM:Logarithmic Over-Parameterization is SufficientNeurIPSN-PyTorch(Author)Image Classification2020
19EagleEye: Fast Sub-net Evaluation for Efficient Neural Network PruningECCVFEagleEyePyTorch(Author)Image Classification2020
20Channel Pruning via Automatic Structure SearchIJCAIFABCPyTorch(Author)Image Classification2020
Pruning After Training CNNs 2019
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Auto-Balanced Filter Pruning for Efficient Convolutional Neural NetworksAAAIF--Image Classification2019
02Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural NetworksNeurIPSFGate DecoratorPyTorch(Author)Image Classification&Semantic Segmentation2019
03EigenDamage: Structured Pruning in the Kronecker-Factored EigenbasisICMLCEigenDamagePyTorch(Author)Image Classification2019
04Importance Estimation for Neural Network PruningCVPRFTaylor-FO-BNPyTorch(Author)Image Classification2019
05The State of Sparsity in Deep Neural NetworksarXivw-TensorFlow(Author)Image Classification&machine translation2019
06Collaborative Channel Pruning for Deep NetworksICMLFCCP-Image Classification2019
07One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizersNeurIPSW--Image Classification2019
08ECC: Platform-Independent Energy-Constrained Deep Neural Network Compression via a Bilinear Regression ModelCVPRFECCPytorch(Author)Image Classification&Semantic Segmentation2019
09Approximated Oracle Filter Pruning for Destructive CNN Width Optimization githubICMLFAOFPPytorch(Author)Image Classification2019
10Sparse Transfer Learning via Winning Lottery TicketsarXivW-PyTorch(Author)Image Classification2019
11Global Sparse Momentum SGD for Pruning Very Deep Neural NetworksNeurIPSW-PyTorch(Author)Image Classification2019
12The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural NetworksICLR (Best)WLTHTensorFlow(Author)Image Classification2019
13Deconstructing Lottery Tickets: Zeros, Signs, and the SupermaskNeurIPSW-TensorFlow(Author)Image Classification2019
14Winning the Lottery with Continuous SparsificationNeurIPSFCSPyTorch(Author)Image Classification2019
15Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated StructureCVPRFC-SGDTensorflow(Author)Image Classification2019
16Exploiting Kernel Sparsity and Entropy for Interpretable CNN CompressionCVPRWKSEPyTorch(Author)Image Classification2019
17Towards Compact ConvNets via Structure-Sparsity Regularized Filter PruningTNNLSFSSRCaffe(Author)Image Classification2019
18Towards Optimal Structured CNN Pruning via Generative Adversarial LearningCVPRFGALPyTorch(Author)Image Classification2019
18Efficient Neural Network CompressionCVPRCENCPyTorch(Author)Image Classification2019
Pruning After Training CNNs 2018
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Accelerating Convolutional Networks via Global & Dynamic Filter PruningIJCAIFGDP-Image Classification2018
02AMC: Automl for model compression and acceleration on mobile devicesECCVFAMCTensorFlow(3rd)Image Classification2018
03Exploring Linear Relationship in Feature Map Subspace for ConvNets CompressionarXivF--Object Detection&Human Pose Estimation2018
04To prune, or not to prune: exploring the efficacy of pruning for model compressionICLRWW-TensorFlow(Author)NLP2018
05CLIP-Q: Deep Network Compression Learning by In-Parallel Pruning-QuantizationCVPRWCLIP-Q-Image Classification2018
06Discrimination-aware Channel Pruning for Deep Neural NetworksNeurIPSCDCPTensorFlow(Author)Image Classification2018
07NISP: Pruning Networks using Neuron Importance Score PropagationCVPRNCNISP-Image Classification2018
082PFPCE: Two-Phase Filter Pruning Based on Conditional EntropyAAAIW2PFPCE-Image Classification2018
Pruning After Training CNNs 2017 and earlier
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Optimal Brain DamageNIPSWOBD-Image Classification1989
02Second Order Derivatives for Network Pruning: Optimal Brain SurgeonNIPSWOBS-Image Classification1992
03Structured Pruning of Deep Convolutional Neural NetworksarXivC--Image Classification2015
04Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman CodingICLR (Best)W-Caffe(Author)Image Classification2016
05ThiNet: A Filter Level Pruning Method for Deep Neural Network CompressionICCV&TPAMIFThiNetCaffe(Author), PyTorch(3rd)Image Classification2017&2019
06Pruning Convolutional Neural Networks for Resource Efficient InferenceICLRF-PyTorchImage Classification2017
07Pruning Filters for Efficient ConvNetsICLRFPFECPyTorch(3rd)Image Classification2017
08Channel pruning for accelerating very deep neural networksICCVC-Caffe(Author)Image Classification&Object Detection2017
1.1.3.2 Pruning ViTs
Pruning After Training ViTs 2024
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Fast and Controllable Post-training Sparsity: Learning Optimal Sparsity Allocation with Global Constraint in MinutesAAAIWFCPTS-Image Classification&Object Detection2024
02UPDP: A Unified Progressive Depth Pruner for CNN and Vision TransformerAAAILUPDP-Image Classification&Object Detection2024
03Pruning Self-attentions into Convolutional Layers in Single PathTPAMIHSPViTPyTorchImage Classification&Object Detection2024
Pruning After Training ViTs 2023
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01X-Pruner: eXplainable Pruning for Vision TransformersCVPRCHX-PrunerPytorch(Author)Image Classification2023
02Global Vision Transformer Pruning with Hessian-Aware SaliencyCVPRCHNViT-Image Classification2023
03Pruning Parameterization with Bi-level Optimization for Efficient Semantic Segmentation on the EdgeCVPRWSTE-semantic Segmentation2023
04Instant Soup: Cheap Pruning Ensembles in A Single Pass Can Draw Lottery Tickets from Large ModelsICMLWISPPytorch(Author)Image Classification&NLP2023
Pruning After Training ViTs 2022
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Width & Depth Pruning for Vision TransformersAAAICWDPruningPytorch(Author)Image Classification2022
02SAViT: Structure-Aware Vision Transformer Pruning via Collaborative OptimizationNeurIPSCHESAViTPytorch(Author)Image Classification&object detection2022
03VTC-LFC: Vision Transformer Compression with Low-Frequency ComponentsNeurIPSCVTC-LFCPytorch(Author)Image Classification2022
04CP-ViT: Cascade Vision Transformer Pruning via Progressive Sparsity PredictionarXivHCP-ViT-Image Classification2022
05Unified Visual Transformer CompressionICLRHUVCPytorch(Author)Image Classification2022
1.1.3.3 Pruning BERTs
Pruning After Training BERTs 2023
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse ApproximationICMLHLoSparsePyTorch(Author)NLP2023
02Instant Soup: Cheap Pruning Ensembles in A Single Pass Can Draw Lottery Tickets from Large ModelsICMLWISPPytorch(Author)Image Classification&NLP2023
03Gradient-Free Structured Pruning with Unlabeled DataICMLFKCM-NLP2023
04The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that MatterarXivW&N:M-Pytorch(Author)NLP2023
Pruning After Training BERTs 2022
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Structured Pruning Learns Compact and Accurate ModelsACLLHCoFiPyTorch(Author)Natural Language Understanding2022
02From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model CompressionAAAIWHCAPPyTorch(Author)NLP2022
03PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight ImportanceICMLWCPLATONPyTorch(Author)Natural Language Understanding&Question Answering&Image Classification2022
04Parameter-Efficient Sparsity for Large Language Models Fine-TuningIJCAIWPSTPyTorch(Author)Language Modeling2022
05The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language ModelsEMNLPWoBERTPyTorch(Author)Natural Language Understanding2022
06Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and PruningNeurIPSWExactOBSPyTorch(Author)Image Classification&Object Detection&Question Answering2022
Pruning After Training BERTs 2021
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Super Tickets in Pre-Trained Language Models: From Model Compression to Improving GeneralizationACLWsuper ticketsPyTorch(Author)Language Understanding2021
02Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable MasksNeurIPSN:MAdaPrunePyTorch(Author)Image Classification2021
03Prune Once for All: Sparse Pre-Trained Language ModelsNeurIPS WorkshopWOFAPyTorch(Author)NLP2021
04BERT Busters: Outlier Dimensions that Disrupt TransformersACLW--NLP2021
05PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech RecognitionNeurIPSWPARP-Speach Recognition2021
06Parameter-Efficient Transfer Learning with Diff PruningACLMDiff PruningPyTorch(Author)NLP2021
07EarlyBERT: Efficient BERT training via early-bird lottery ticketsACL-IJCNLPHEarlyBERTPyTorch(Author)NLP2021
08The Lottery Ticket Hypothesis for Pre-trained BERT NetworksICMLW-PyTorch(Author)Language Modeling2021
09Structured Pruning of Large Language ModelsarXivWFLOPPyTorch(Author)NLP classification2021
10Accelerating Sparse Deep Neural NetworksarXivW--Image Classification&Image Segmentation and Detection&Language Modeling&Language Translation2021
11Differentiable Subset Pruning of Transformer HeadsTACLH-PyTorch(Author)NLP2021
Pruning After Training BERTs 2020
No.TitleVenueTypeAlgorithm NameCodeAPPYear
03Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of TransformersICMLW--NLP2020
04When BERT Plays the Lottery, All Tickets Are WinningEMNLPW-PyTorch(Author)Language Modeling2020
05LadaBERT: Lightweight Adaptation of BERT through Hybrid Model CompressionCOLINGW--NLP(Sentiment Classification,Natural Language Inference,Pairwise Semantic Equivalence)2020
06Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity PriorEMNLPH--NLP2020
07Compressing BERT: Studying the Effects of Weight Pruning on Transfer LearningRep4NLPW--NLP2020
Pruning After Training BERTs 2019
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Reweighted Proximal Pruning for Large-Scale Language RepresentationarXivOther--NLP2019
02Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured PruningEMNLPOther--NLP2019
1.1.3.4 Pruning LLMs
Pruning After Training LLMs 2024
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-TuningACLCHLoRAPrunePyTorch(Author)Language Modeling&Classification2024
02A Simple and Effective Pruning Approach for Large Language ModelsICLRWWandaPyTorch(Author)Language Modeling&Classification2024
03SliceGPT: Compress Large Language Models by Deleting Rows and ColumnsICLRCHSliceGPTPyTorch(Author)Language Modeling&Classification2024
04Fluctuation-based Adaptive Structured Pruning for Large Language ModelsAAAICHFLAPPyTorch(Author)Language Modeling&Classification2024
05BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity AllocationarXivBBESAPyTorch(Author)Language Modeling&Classification2024
06APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and InferenceICMLHCAPTPyTorch(Author)Classification2024
07Sheared LLaMA: Accelerating Language Model Pre-training via Structured PruningICLRCHSheared LLaMAPyTorch(Author)Language Modeling&Classification2024
08Everybody Prune Now: Structured Pruning of LLMs with only Forward PassesarXivCHBonsaiPyTorch(Author)Language Modeling&Classification2024
09LaCo: Large Language Model Pruning via Layer CollapsearXivLLaCo-Language Modeling&Classification2024
10ShortGPT: Layers in Large Language Models are More Redundant Than You ExpectarXivLShortGPT-Language Modeling&Classification2024
11SparseLLM: Towards Global Pruning for Pre-trained Language ModelsarXivBSparseLLMPyTorch(Author)Language Modeling&Classification2024
12SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer BlocksarXivNSLEBPyTorch(Author)Language Modeling&Classification2024
13Streamlining Redundant Layers to Compress Large Language ModelsarXivLLLMStreamline-Language Modeling&Classification2024
14Why Lift so Heavy? Slimming Large Language Models by Cutting Off the LayersarXivL--Classification2024
15Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining MethodsICLRWHC-PyTorch(Author)Classification2024
16Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured SparsityVLDBWFlash-LLMPyTorch(Author)Recognizing Textual Entailment2024
17The LLM SurgeonarXivWCLLM SurgeonPyTorch(Author)Language Modeling2024
18Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High SparsityICMLWOWLPyTorch(Author)Language Modeling&Classification2024
19The Unreasonable Ineffectiveness of the Deeper LayersarXivB--Classification2024
20Enhancing One-Shot Pruned Generative Pre-training Language Models through Sparse-Dense-Sparse MechanismOpenReviewWSDS-Classification2024
21KS-Lottery: Finding Certified Lottery Tickets for Multilingual Language ModelsarXivW--Language Translation2024
Pruning After Training LLMs 2023
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01SparseGPT: Massive Language Models Can be Accurately Pruned in One-ShotNeurIPSWP-PyTorch(Author)Language Modeling&Classification2023
02LLM-Pruner: On the Structural Pruning of Large Language ModelsarXivCHPLLM-PrunerPyTorch(Author)Language Modeling&Language Generation&Classification2023
03LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge RecoveryarXivCHLoRAShear-Language Modeling&Language Generation&Classification2023
04Compresso: Structured Pruning with Collaborative Prompting Learns Compact Large Language ModelsarXivCHCompressoPyTorch(Author)Classification2023
05Mini-GPTs: Efficient Large Language Models through Contextual PruningarXivWC--Language Modeling& Classification2023
06The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that MatterarXivW&N:M-Pytorch(Author)NLP2023
1.1.3.5 Pruning Diffusion Models
Pruning After Training Diffusion Models 2023
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Structural Pruning for Diffusion ModelsNeurIPSCDiff-PruningPyTorch(Author)Image Generation2023
1.1.3.6 Pruning Vision-and-Languages
Pruning After Training VLMs 2024
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language ModelsICLRLECoFLaPPytorch(Author)VQA&Image Captioning&Image-text Retrieval&Image Classification2024
Pruning After Training VLMs 2023
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Large Multimodal Model Compression via Efficient Pruning and Distillation at AntGrouparXivB--Multimodal Advertisement Audition2023
02UPop: Unified and Progressive Pruning for Compressing Vision-Language TransformersICMLHUPopPytorch(Author)Image Classification&Image Caption&Image Retrieval&VQA2023
03Instant Soup: Cheap Pruning Ensembles in A Single Pass Can Draw Lottery Tickets from Large ModelsICMLWISPPytorch(Author)Image Classification&NLP2023
Pruning After Training VLMs 2022
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Playing Lottery Tickets with Vision and LanguageAAAIW--Vision-and-Language2022
1.1.3.7 Pruning Other Models
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be PrunedACLW-PyTorch(Author)NLP2019
02Playing the Lottery with Rewards and Multiple Languages: Lottery Tickets in RL and NLPICLRW--Classic Control&Atari Game2020
03Dynamic Sparsity Neural Networks for Automatic Speech RecognitionICASSPW--Speach Recognition2021
04GAN Compression: Efficient Architectures for Interactive Conditional GANsarXivC--Image-to-Image Translation2021
05Content-Aware GAN CompressionCVPRF-PyTorch(Author)Image Generation, Image Projection, Image Editing
06A Unified Lottery Ticket Hypothesis for Graph Neural NetworksICMLW-PyTorch(Author)Node Classification&Link Prediction2021
07Winning Lottery Tickets in Deep Generative ModelsAAAIW--Image generative2021
08GANs Can Play Lottery Tickets TooICLRW-PyTorch(Author)Image generative2021
09Layer-wise Pruning of Transformer Attention Heads for Efficient Language ModelingarXivH-PyTorch(Author)Lanugage Modeling2021
10Can We Find Strong Lottery Tickets in Generative Models?arXivW--Image generative2022
11Exploring Lottery Ticket Hypothesis in Spiking Neural NetworksECCVWETPyTorch(Author)Image Classification2022
12Structured Pruning for Efficient Generative Pre-trained Language ModelsACLCCP3-Language Modeling&Machine Translation&Abstractive Summarization2023
13Rethinking Graph Lottery Tickets: Graph Sparsity MattersICLRW--Node Classification2023
14CP3: Channel Pruning Plug-in for Point-based NetworksCVPRCCP3-3D Image Classification and Object Detection2023
1.1.3.8 Post Training
Post Training 2024
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Fast and Controllable Post-training Sparsity: Learning Optimal Sparsity Allocation with Global Constraint in MinutesAAAIWFCPTS-Image Classification2024
Post Training 2023
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01SparseGPT: Massive Language Models Can be Accurately Pruned in One-ShotNeurIPSWP-PyTorch(Author)Language Modeling2023
02Unified Data-Free Compression: Pruning and Quantization without Fine-TuningICCVCUDFC-Image Classification2023
03OTOv3: Automatic Architecture-Agnostic Neural Network Training and Compression from Structured Pruning to Erasing OperatorsarXivWFC--Image Classification2023
Post Training 2022
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01CP-ViT: Cascade Vision Transformer Pruning via Progressive Sparsity PredictionarXivHCP-ViT-Image Classification2022
02Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and PruningNeurIPSWExactOBSPyTorch(Author)Image Classification&Object Detection&Question Answering2022
03A Fast Post-Training Pruning Framework for TransformersNeurIPSHF-PyTorch(Author)Natural Language Understanding2022
Post Training 2021
No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Enabling Retrain-free Deep Neural Network Pruning Using Surrogate Lagrangian RelaxationIJCAIW--Image Classification & Object Detection2021
02Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable MasksNeurIPSN:MAdaPrunePyTorch(Author)Image Classification2021

1.1.4 Pruning in Early Training

No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Linear Mode Connectivity and the Lottery Ticket HypothesisICMLW--Image Classification2020
02When To Prune? A Policy Towards Early Structural PruningCVPRFPaT-Image Classification2022
03Drawing Early-Bird Tickets: Towards More Efficient Training of Deep NetworksICLRW-PyTorch(Author)Image Classification2020
04A Gradient Flow Framework For Analyzing Network PruningICLRF-PyTorch(Author)Image Classification2021

1.2 Dynamic Pruning

No.TitleVenueTypeAlgorithm NameCodeAPPYear
01Channel Gating Neural NetworksNeurIPSFRNP-Image Classification2017
02Channel Gating Neural NetworksNeurIPSCCGNetPyTorch(Author)Image Classification2019
03Dynamic Channel Pruning: Feature Boosting and SuppressionICLRCFBSPyTorch(Author)Image Classification2019
04Frequency-Domain Dynamic Pruning for Convolutional Neural NetworksNeurIPSFFDNP-Image Classification2019
05Fire Together Wire Together: A Dynamic Pruning Approach With Self-Supervised Mask PredictionCVPRF--Image Classification2019
06Dynamic Dual Gating Neural NetworksICCVCDGNetPyTorch(Author)Image Classification2021
07Manifold Regularized Dynamic Network PruningCVPRFManiDPPyTorch(Author)Image Classification2021
08Contrastive Dual Gating: Learning Sparse Features With Contrastive LearningCVPRWFCDG-Image Classification2022

2. Learning and Pruning

2.1 Continual learning

No.TitleVenueAlgorithm NameCodeAPPYear
01Continual Learning via Neural PruningarXivCLNP-Image Classification2019
02Learning Bayesian Sparse Networks With Full Experience Replay for Continual LearningCVPRSNCL-Image Classification2022
03Continual Prune-and-Select: Class-Incremental Learning with SPecialized SubnetworksApplied Intelligence-PyTorch(Author)Image Classification2023
04Continual Domain Adaptation through Pruning-aided Domain-specific Weight ModulationCVPRWPaCDAPyTorch(Author)Image Classification2023

2.2 Contrastive learning

No.TitleVenueAlgorithm NameCodeAPPYear
01Studying the impact of magnitude pruning on contrastive learning methodsICML-PyTorch(Author)Image Classification2020
02Training Debiased Subnetworks with Contrastive Weight PruningCVPRDCWP-Image Classification2023

2.3 Federated learning

No.TitleVenueAlgorithm NameCodeAPPYear
01FedDUAP: Federated Learning with Dynamic Update and Adaptive Pruning Using Shared Data on the ServerIJCAIFedDUAP-Image Classification2020
02Model Pruning Enables Efficient Federated Learning on Edge DevicesTNNLS-PyTorch(Author)Image Classification2022

3. Application

3.1 Computer Vision

No.TitleVenueCodeAPPYear
01Deep Rewiring: Training very Sparse Deep NetworksICLR-Image Classification&Audio2018
02Co-Evolutionary Compression for Unpaired Image TranslationICCVPyTorch(Author)Image Style Translation2019
03Content-Aware GAN CompressionCVPRPyTorch(Author)Image Style Translation2021
04Training Neural Networks with Fixed Sparse MasksNeurIPSPyTorch(Author)Image Classification2021
05Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization SpaceCVPRPyTorch(Author)Image Classification&Audio2022
06SuperTickets: Drawing Task-Agnostic Lottery Tickets from Supernets via Jointly Architecture Searching and Parameter PruningECCVPyTorch(Author)Image Classification&Object Detection&Human Pose Estimation2022

3.2 Natural Language Processing

No.TitleVenueCodeAPPYear
01When BERT Plays the Lottery, All Tickets Are WinningEMNLPPyTorch(Author)Language Modeling2020
02The Lottery Ticket Hypothesis for Pre-trained BERT NetworksICMLPyTorch(Author)Language Modeling2021
03Structured Pruning Learns Compact and Accurate ModelsACLPyTorch(Author)Natural Language Understanding2022
04A Fast Post-Training Pruning Framework for TransformersNeurIPSPyTorch(Author)Natural Language Understanding2022
05A Fast Post-Training Pruning Framework for TransformersNeurIPSPyTorch(Author)Natural Language Understanding2022
06The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language ModelsEMNLPPyTorch(Author)Natural Language Understanding2022
07Pruning Meets Low-Rank Parameter-efficientarXiv-Image Classification&Language Modeling2023
08LLM-Pruner: On the Structural Pruning of Large Language ModelsarXiv-Language Modeling2023

3.3 Audio Signal Processing

No.TitleVenueCodeAPPYear
01Exploring Sparsity in recurrent neural networksICLRPyTorchSpeech Recognition2017
02Deep Rewiring: Training very Sparse Deep NetworksICLR-Image Classification&Audio2018

4. Combination

4.1 Pruning and Quantization

No.TitleVenueCodeAPPYear
01CLIP-Q: Deep Network Compression Learning by In-Parallel Pruning-QuantizationCVPR-Image Classification2018
02Accelerating Sparse Deep Neural NetworksarXiv-Image Classification&Object Detection&Language Translation&Language Modeling&Image Synthesis&Domain Translation&Style Transfer&Image-Image Translation&Super Resolution2021
03OPQ: Compressing Deep Neural Networks with One-shot Pruning-QuantizationAAAI-Image Classification2021
04Deep Model Compression Based on the Training HistoryarXiv-Image Classification2022
05LLM-Pruner: On the Structural Pruning of Large Language ModelsarXivPyTorchCausal Language Modeling2023
06Unified Data-Free Compression: Pruning and Quantization without Fine-TuningICCV-Image Classification2023

5. Survey of Pruning

Survey of Pruning 2024

No.TitleVenueCodeAPPYear
01Structured Pruning for Deep Convolutional Neural Networks: A surveyTPAMI-CV&NLP2024
02A survey on efficient vision transformers: algorithms, techniques, and performance benchmarkingarXiv-CV2024
03A Survey of Lottery Ticket HypothesisarXiv-CV&NLP2024
04Model Compression and Efficient Inference for Large Language Models: A SurveyarXiv-NLP2024

Survey of Pruning 2023

No.TitleVenueCodeAPPYear
01Why is the State of Neural Network Pruning so Confusing? On the Fairness, Comparison Setup, and Trainability in Network PruningarXivPyTorch(Author)Image Classification2023
02Transforming Large-Size to Lightweight Deep Neural Networks for IoT ApplicationsACM Computing Surveys-CV&NLP&Audio2023
03A Survey on Model Compression for Large Language ModelsTACL-NLP&Unseen Instructions2023
04Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to SystemsarXiv--2023
05A Survey on Dynamic Neural Networks for Natural Language ProcessingarXiv-NLP2023
06Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a SurveyarXiv-CV&NLP2023

Survey of Pruning 2022

No.TitleVenueCodeAPPYear
01A Survey on Efficient Convolutional Neural Networks and Hardware AccelerationElectronics--2022
02Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a SurveyarXiv-Image Classification2022
03Efficient Transformers: A SurveyarXiv-CV&NLP2022
04Recent Advances on Neural Network Pruning at InitializationIJCAI-CV&NLP2022

Survey of Pruning 2021

No.TitleVenueCodeAPPYear
01Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networksJMLR-Image Classification2021
02Dynamic Neural Networks: A SurveyarXiv--2021
03Pruning and Quantization for Deep Neural Network Acceleration: A SurveyNeurocomputing-Image Classification2021
04Compressing Large-Scale Transformer-Based Models: A Case Study on BERTTACL-NLP2021

Survey of Pruning 2020

No.TitleVenueCodeAPPYear
01Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive SurveyIEEE--2020
02Pruning Algorithms to Accelerate Convolutional Neural Networks for Edge Applications: A SurveyarXiv-Image Classification2020
03A Survey of Model Compression and Acceleration for Deep Neural NetworksarXiv--2020
04An Survey of Neural Network CompressionarXiv--2020
05Convolutional Neural Network Pruning: A SurveyCCC--2020
06What is the State of Neural Network Pruning?MLSys--2020
07A comprehensive survey on model compression and accelerationArtificial Intelligence Review--2020
08A Survey on Deep Neural Network Compression: Challenges, Overview, and SolutionsarXiv--2020

Survey of Pruning 2019 and earlier

No.TitleVenueCodeAPPYear
01Pruning Algorithms-A SurveyIEEE Transactions on Neural Networks-Image Classification1993
02Efficient Processing of Deep Neural Networks: A Tutorial and SurveyarXiv-Image Classification2017
03Recent advances in efficient computation of deep convolutional neural networksarXiv--2018
04The State of Sparsity in Deep Neural NetworksarXivPyTorch(Author)Image Classification&machine translation2019

6. Other Works

Papers

No.TitleVenueAlgorithm NameCodeAPPYear
01Is Pruning Compression?: Investigating Pruning Via Network Layer SimilarityWACV--Image Classification2020
02A Gradient Flow Framework For Analyzing Network PruningICLR-PyTorch(Author)Image Classification2021
03Data Level Lottery Ticket Hypothesis for Vision TransformersIJCAI-PyTorch(Author)Image Classification2021
04Are All Layers Created Equal?JMLR--Image Classification2022

https://github.com/airaria/TextPruner

Acknowledgements

We would like to express our gratitude to the authors of the articles cited in our survey and the authors of the following repositories.

https://github.com/he-y/awesome-Pruning/
https://github.com/MingSun-Tse/Awesome-Pruning-at-Initialization
https://github.com/csyhhu/Awesome-Deep-Neural-Network-Compression/blob/master/Paper/Pruning.md

Citation

If you find this project useful, please cite

@article{cheng2023survey,
  title={A Survey on Deep Neural Network Pruning:Taxonomy, Comparison, Analysis, and Recommendations},
  author={Hongrong Cheng and Miao Zhang and Javen Qinfeng Shi},
  journal={arXiv preprint arXiv:2308.06767},
  year={2023}
}

Similar repos

ghimiredhikura/Awasome-Pruning

Awesome Pruning. âś… Curated Resources for Neural Network Pruning.

178

MingSun-Tse/Efficient-Deep-Learning

Collection of recent methods on (deep) neural network compression and acceleration.

958

liyunqianggyn/Awesome-LLMs-Pruning

Awesome LLM pruning papers all-in-one repository with integrating all useful resources and…

180

VainF/Torch-Pruning

[CVPR 2023] DepGraph: Towards Any Structural Pruning; LLMs, Vision Foundation Models, etc.

3,357

cedrickchee/awesome-ml-model-compression

Awesome machine learning model compression research papers, quantization, tools, and learning…

546

horseee/Awesome-Efficient-LLM

A curated list for Efficient Large Language Models

2,043

Zhen-Dong/Awesome-Quantization-Papers

List of papers related to neural network quantization in recent AI conferences and journals.

853

windmaple/awesome-AutoML

Curating a list of AutoML-related research, tools, projects and other resources

944

Contributors

hrcheng1066

340 commits