vantienpham/Awesome-Tensor-Decomposition

😎 A curated list of tensor decomposition resources for model compression.

121

248 commits

updated Sep 20, 2026

See the code

README

Awesome Tensor Decomposition

A curated list of tensor decomposition resources for model compression.

📋 Research Papers

Transformer, LLM & more

TitleVenueYear
Understanding Calibration and Truncation Error Propagation in Training-Free Low-Rank Compression for LLMsCOLM2026
Surviving Resource-constraint Compression: Capability Retention under Tensor-train Decomposition for Sub-billion Parameter Language ModelsMachine Intelligence Research2026
Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression GitHub Repo starsICML2026
Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression GitHub Repo starsICML2026
Advancing SVD-based LLM Compression via Layer-Wise Error Model Search GitHub Repo starsICML2026
CARE: Covariance-Aware and Rank-Enhanced Decomposition for Enabling Multi-Head Latent Attention GitHub Repo starsICLR2026
Scalable Kronecker-Factored Fisher Approximation for Neural Network Parameter Sensitivity GitHub Repo starsICLR2026
LeSTD: LLM Compression via Learning-based Sparse Tensor DecompositionICLR2026
Large Language Model Compression with Global Rank and Sparsity OptimizationICLR2026
UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMsICLR2026
WSVD: Weighted Low-Rank Approximation for Fast and Efficient Execution of Low-Precision Vision-Language ModelsICLR2026
TD-MoE: Tensor Decomposition for MoE ModelsICLR2026
Efficient Resource-Constrained Training of Vision Transformers via Subspace OptimizationICLR2026
Theoretical guarantees for low-rank compression of deep neural networksApplied and Computational Harmonic Analysis2026
FLAT-LLM: Fine-grained Low-rank Activation Space Transformation for Large Language Model Compression GitHub Repo starsFindings of the Association for Computational Linguistics2026
QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Models GitHub Repo starsNeurIPS2025
MoE-SVD: Structured Mixture-of-Experts LLMs Compression via Singular Value DecompositionICML2025
LS-PRISM: A layer-selective pruning method via low-rank approximation and sparsification for efficient large language model compressionNeural Networks2025
Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device LearningICML2025
Canonical Rank Adaptation: An Efficient Fine-Tuning Strategy for Vision Transformers GitHub Repo starsICML2025
SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression GitHub Repo starsAAAI2025
SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression GitHub Repo starsNAACL2025
NdLinear Is All You Need for Representation Learning GitHub Repo starsArxiv2025
xKV: Cross-Layer SVD for KV-Cache Compression GitHub Repo starsArxiv2025
You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning GitHub Repo starsICLR2025
SliceGPT: Compress Large Language Models by Deleting Rows and Columns GitHub Repo starsICLR2025
Efficient Learning With Sine-Activated Low-rank MatricesICLR2025
Palu: Compression KV-Cache with Low-Rank Decomposition GitHub Repo starsICLR2025
Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives GitHub Repo starsICLR2025
MoDeGPT: Modular Decomposition for Large Language Model CompressionICLR2025
Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression GitHub Repo starsICLR2025
SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression GitHub Repo starsICLR2025
HASSLE-free: A unified Framework for Sparse plus Low-Rank Matrix Decomposition for LLMs GitHub Repo starsCPAL2025
FLAR-SVD: Fast and Latency-Aware Singular Value Decomposition for Model Compression GitHub Repo starsCVPRW2025
Breaking the Low-Rank Dilemma of Linear Attention GitHub Repo starsCVPR2025
FlexiGPT: Pruning and Extending Large Language Models with Low-Rank Weight SharingNAACL2025
TensorLLM: Tensorising Multi-Head Attention for Enhanced Reasoning and Compression in LLMsArxiv2025
Memory-Efficient Vision Transformers: An Activation-Aware Mixed-Rank Compression StrategyECCVW2024
Searching for Efficient Linear Layers over a Continuous Space of Structured Matrices GitHub Repo starsNeurIPS2024
DBA: Efficient Transformer With Dynamic Bilinear Low-Rank AttentionTNNLS2024
Tensor Product Attention Is All You Need GitHub Repo starsArxiv2025
EoRA: Training-free Compensation for Compressed LLM with Eigenspace Low-Rank ApproximationArxiv2024
Compressing Large Language Models using Low Rank and Low Precision Decomposition GitHub Repo starsNeurIPS2024
ESPACE: Dimensionality Reduction of Activations for Model Compression GitHub Repo starsNeurIPS2024
MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition GitHub Repo starsEMNLP 2024 Findings2024
Tensor Train Low-rank Approximation (TT-LoRA): Democratizing AI with Accelerated LLMsArxiv2024
Dual-grained Lightweight StrategyTPAMI2024
Adaptive Feature-based Low-Rank Compression of Large Language Models via Bayesian OptimizationEMNLP findings2024
Geometry is All You Need: A Unified Taxonomy of Matrix and Tensor Factorization for Compression of Generative Language ModelsArxiv2024
From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients GitHub Repo starsArxiv2024
The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction GitHub Repo starsICLR2024
ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models GitHub Repo starsArxiv2024
AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuning GitHub Repo starsEMNLP2024
Adaptive Rank Selections for Low-Rank Approximation of Language Models GitHub Repo starsNAACL2024
Dynamic Low-rank Estimation for Transformer-based Language ModelsNAACL Findings2024
PELA: Learning Parameter-Efficient Models with Low-Rank Approximation GitHub Repo starsCVPR2024
Adaptive Rank Selections for Low-Rank Approximation of Language ModelsACL2024
TRAWL: Tensor Reduced and Approximated Weights for Large Language ModelsArxiv2024
Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantization GitHub Repo starsISCA2024
Unified Low-rank Compression Framework for Click-through Rate Prediction GitHub Repo starsKDD2024 ADS2024
LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models GitHub Repo starsNAACL2024
CoMERA: Computing- and Memory-Efficient Training via Rank-Adaptive Tensor OptimizationArxiv2024
FLORA: Fine-grained Low-Rank Architecture Search for Vision Transformer GitHub Repo starsWACV2024
FLoRA: Low-Rank Core Space for N-dimension GitHub Repo starsArxiv2024
Enhancing GAN Performance Through Neural Architecture Search and Tensor Decomposition GitHub Repo starsICASSP2024
A Computational Study of Matrix Decomposition Methods for Compression of Pre-trained Transformers GitHub Repo starsPACLIC2024
Singularformer: Learning to Decompose Self-Attention to Linearize the Complexity of Transformer GitHub Repo starsIJCAI2023
LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation GitHub Repo starsICML2023
Singularformer: Learning to Decompose Self-Attention to Linearize the Complexity of Transformer GitHub Repo starsIJCAI2023
FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer GitHub Repo starsAAAI2023
LightFormer: Light-weight Transformer Using SVD-based Weight Transfer and Parameter SharingACL2023
Low-Rank Updates of pre-trained Weights for Multi-Task LearningACL2023
Compressing Transformers: Features Are Low-Rank, but Weights Are Not!AAAI2023
TT-MLP: Tensor Train Decomposition on Deep MLPsIEEE Access2023
Matrix Compression via Randomized Low Rank and Low Precision Factorization GitHub Repo starsNeurIPS2023
Strategies for Applying Low Rank Decomposition to Transformer-Based ModelsNeurIPS2022
Hypoformer: Hybrid Decomposition Transformer for Edge-friendly Neural Machine TranslationEMNLP2022
Kronecker Decomposition for GPT CompressionACL2022
Deeply Tensor Compressed Transformers for End-to-End Object DetectionAAAI2022
Language model compression with weighted low-rank factorization GitHub Repo starsICLR2022
DRONE: Data-aware Low-rank Compression for Large NLP ModelsNeurIPS2021
Linformer: Self-Attention with Linear Complexity GitHub Repo starsArxiv2020
Lightweight and Efficient End-to-End Speech Recognition Using Low-Rank Transformer GitHub Repo stars GitHub Repo starsICASSP2020
A Tensorized Transformer for Language Modeling GitHub Repo starsNeurIPS2019
GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model ShrinkingNIPS2018

CNN

TitleVenueYear
Basis Sharing and Coefficient Pruning for Efficient Neural Network CompressionNeural Networks2026
A Novel Approach to Lossless Convolutional Neural Network Compression via Progressive Knowledge Distillation-Incorporated Low-Rank CompressionNeural Networks2026
Deep Neural Network Compression via Data-Driven Low-Rank Singular Value DecompositionJournal of Artificial Intelligence Research2026
Adaptive Tucker Decomposition-based Progressive Model Compression for Convolutional Neural NetworksExpert Systems with Applications2026
Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networks GitHub Repo starsNeurIPS2025
Coupled Tensor Decomposition for Compact Network Representation GitHub Repo starsTNNLS2025
Reduced storage direct tensor ring decomposition for convolutional neural networks compression GitHub Repo starsNeural Networks2025
Co-Exploring Structured Sparsification and Low-Rank Tensor Decomposition for Compact DNNsTNNLS2025
Rope-net: deep convolutional neural network via robust principal component analysisMachine Learning2025
QuaDCNN: Quantized compression of deep CNN based on tensor-train decomposition with automatic rank determinationNeurocomputing2025
An overview of tensor and matrix decomposition methods, applied to deep neural networksEvolving Systems2025
FETTA: Flexible and Efficient Hardware Accelerator for Tensorized Neural Network TrainingArxiv2025
Fine-grained hierarchical singular value decomposition for convolutional neural networks compression and accelerationNeurocomputing2025
Unified Framework for Pre-trained Neural Network Compression via Decomposition and Optimized Rank SelectionECML PKDD2025
Low-Rank Compression for IMC ArraysArxiv2025
Enhancing GANs With MMD Neural Architecture Search, PMish Activation Function, and Adaptive Rank Decomposition GitHub Repo starsIEEE ACCESS2024
Quantization Aware Factorization for Deep Neural Network CompressionJAIR2024
J-Tucker: Joint Compression Scheme for Efficient Deployment of Multi-Task Deep Learning Models on Edge DevicesIEEE Network2024
3D-KCPNet: Efficient 3DCNNs based on tensor mapping theoryNeurocomputing2024
Fully Tensorized Lightweight ConvLSTM Neural Networks for Hyperspectral Image ClassificationTNNLS2024
TEC-CNN: Towards Efficient Compressing Convolutional Neural Nets with Low-rank Tensor DecompositionTOMM2024
Activation Map Compression through Tensor Decomposition for Deep Learning GitHub Repo starsNeurIPS2024
Geometry-aware training of factorized layers in tensor Tucker formatNeurIPS2024
Robustness of Tensor Decomposition-Based Neural Network CompressionICIP2024
Stable Low-Rank CP Decomposition for Compression of Convolutional Neural Networks Based on SensitivityApplied Sciences2024
How to Train Your Unstable Looped Tensor NetworkJSTSP2024
Learning Low-Rank Tensor Cores with Probabilistic l0-Regularized Rank Selection for Model Compression GitHub Repo starsIJCAI2024
Structure-Preserving Network Compression Via Low-Rank Induced Training Through Linear Layers Composition GitHub Repo starsTMLR2024
Compact Model Training by Low-Rank Projection With Energy Transfer GitHub Repo starsTNNLS2024
An Accuracy-Preserving Neural Network Compression via Tucker DecompositionIEEE Transactions on Sustainable Computing2024
Convolution Filter Compression via Sparse Linear Combinations of Quantized BasisTNNLS2024
Coarse-To-Fine Tensor Trains for Compact Visual Representations GitHub Repo starsICML2024
Position: Tensor Networks are a Valuable Asset for Green AIICML2024
Compression-aware Training of Neural Networks using Frank-Wolfe GitHub Repo starsArxiv2024
A Practical Approach for Employing Tensor Train Decomposition in Edge DevicesIJPP2024
Neural Network Compression Based on Tensor Ring DecompositionTNNLS2024
Enhanced network compression through tensor decompositions and pruning GitHub Repo starsTNNLS2024
Deep Convolutional Neural Network Compression Method: Tensor Ring Decomposition with Variational Bayesian ApproachNeural Processing Letters2024
Deep Learning Model Compression With Rank Reduction in Tensor DecompositionTNNLS2023
MARS: Masked Automatic Ranks Selection in Tensor Decompositions GitHub Repo starsAISTATS2023
Mixed-TD: Efficient Neural Network Accelerator with Layer-Specific Tensor Decomposition GitHub Repo starsFPL2023
SVD-NAS: Coupling Low-Rank Approximation and Neural Architecture Search GitHub Repo starsWACV2023
How Informative is the Approximation Error from Tensor Decomposition for Neural Network Compression?ICLR2023
Learning Low-Rank Representations for Model CompressionIJCNN2023
Tensor shape search for efficient compression of tensorized data and neural networksApplied Soft Computing2023
Compressing convolutional neural networks with hierarchical Tucker-2 decomposition GitHub Repo starsApplied Soft Computing2023
Tensor shape search for efficient compression of tensorized data and neural networksApplied Soft Computing2023
An effective low-rank compression with a joint rank selection followed by a compression-friendly trainingNeural Networks2023
Joint matrix decomposition for deep convolutional neural networks compression GitHub Repo starsNeurocomputing2023
Training Acceleration of Low-Rank Decomposed Networks using Sequential Freezing and Rank QuantizationArxiv2023
Knowledge Transfer via Decomposing Essential Information in Convolutional Neural Networks GitHub Repo starsTNNLS2022
Compression of Deep Neural Networks based on quantized tensor decomposition to implement on reconfigurable hardware platformsNeural Networks2022
Teacher–student knowledge distillation based on decomposed deep feature representation for intelligent mobile applicationsExpert Systems with Applications2022
HODEC: Towards Efficient High-Order DEcomposed Convolutional Neural NetworksCVPR2022
Towards Practical Control of Singular Values of Convolutional Layers GitHub Repo starsNeurIPS2022
Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations GitHub Repo starsNeurIPS2022
Realistic acceleration of neural networks with fine-grained tensor decompositionNeurocomputing2022
BATUDE: Budget-Aware Neural Network Compression Based on Tucker DecompositionAAAI2022
Convolutional Neural Network Compression through Generalized Kronecker Product Decomposition GitHub Repo starsAAAI2022
Towards Compact Neural Networks via End-to-End Training: A Bayesian Tensor Approach with Automatic Rank Determination GitHub Repo starsSIMODS2022
A novel compact design of convolutional layers with spatial transformation towards lower-rank representation for image classification GitHub Repo starsKnowledge-Based Systems2022
Deep neural network compression by Tucker decomposition with nonlinear responseKnowledge-Based Systems2022
Nested compression of convolutional neural networks with Tucker-2 decompositionIJCNN2022
PSM-nets: Compressing Neural Networks with Product of Sparse MatricesIJCNN2022
PARS: Proxy-Based Automatic Rank Selection for Neural Network Compression via Low-Rank Weight ApproximationMathematics2022
A Design Space Exploration Methodology for Enabling Tensor Train Decomposition in Edge DevicesSAMOS2022
Spatiotemporal Joint Filter Decomposition in 3D Convolutional Neural NetworksNeurIPS2021
Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition GitHub Repo starsNeurIPS2021
Deeply Shared Filter Bases for Parameter-Efficient Convolutional Neural Networks GitHub Repo starsNeurIPS2021
Decomposable-Net: Scalable Low-Rank Compression for Neural Networks GitHub Repo starsIJCAI2021
Tensor Regression Networks GitHub Repo starsJMLR2021
Parameter Efficient Dynamic Convolution via Tensor Decomposition GitHub Repo starsBMVC2021
Nonlinear tensor train format for deep neural network compressionNeural Networks2021
Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization FrameworkCVPR2021
Pufferfish: Communication-efficient Models At No Extra Cost GitHub Repo starsMLSys2021
Learning-based Tensor Decomposition with Adaptive Rank Penalty for CNNs CompressionMIPR2021
Deep Convolutional Neural Network Compression via Coupled Tensor DecompositionJSTSP2021
Tensor Reordering for CNN CompressionICASSP2021
Block-term tensor neural networksNeural Networks2020
Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural NetworkECCV2020
Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer GitHub Repo starsCVPR2020
Few Sample Knowledge Distillation for Efficient Network Compression GitHub Repo starsCVPR2020
Factorized Higher-Order CNNs with an Application to Spatio-Temporal Emotion EstimationCVPR2020
Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification GitHub Repo starsCVPRW2020
T-Basis: a Compact Representation for Neural Networks GitHub Repo starsICML2020
PENNI: Pruned Kernel Sharing for Efficient CNN Inference GitHub Repo starsICML2020
A Novel Rank Selection Scheme in Tensor Ring Decomposition Based on Reinforcement Learning for Deep Neural NetworksICASSP2020
Holistic CNN Compression via Low-Rank Decomposition with Knowledge Transfer GitHub Repo starsTPAMI2019
LTNN: A Layerwise Tensorized Compression of Multilayer Neural NetworkTNNLS2019
Einconv: Exploring Unexplored Tensor Network Decompositions for Convolutional Neural Networks GitHub Repo starsNeurIPS2019
Efficient Neural Network Compression GitHub Repo starsCVPR2019
ADA-Tucker: Compressing deep neural networks via adaptive dimension adjustment tucker decompositionNeural Networks2019
RotDCF: Decomposition of Convolutional Filters for Rotation-Equivariant Deep Networks GitHub Repo starsICLR2019
Learning Filter Basis for Convolutional Neural Network Compression GitHub Repo starsICCV2019
Automated Multi-Stage Compression of Neural Networks GitHub Repo starsICCVW2019
Compressing Deep Models using Multi Tensor Train DecompositionICCAIS2019
Compressing Fully Connected Layers using Kronecker Tensor DecompositionICCSNT2019
Adaptive Mixture of Low-Rank Factorizations for Compact Neural Modeling GitHub Repo starsOpenReview2019
Tensor-factorized neural networksTNNLS2018
Sharing Residual Units Through Collective Tensor Factorization To Improve Deep Neural NetworksIJCAI2018
Learning Compact Recurrent Neural Networks with Block-Term Tensor DecompositionCVPR2018
Wide Compression: Tensor Ring NetsCVPR2018
DCFNet: Deep Neural Network with Decomposed Convolutional Filters GitHub Repo starsICML2018
Self-supervised Knowledge Distillation Using Singular Value Decomposition GitHub Repo starsECCV2018
Extreme Network Compression via Filter Group ApproximationECCV2018
Network Decoupling: From Regular to Depthwise Separable Convolutions GitHub Repo starsBMVC2018
On Compressing Deep Models by Low Rank and Sparse DecompositionCVPR2017
Coordinating Filters for Faster Deep Neural Networks GitHub Repo starsICCV2017
Factorized Convolutional Neural NetworksICCVW2017
Tensor Regression Networks with various Low-Rank Tensor Approximations GitHub Repo starsArxiv2017
Accelerating Very Deep Convolutional Networks for Classification and DetectionTPAMI2016
Convolutional Neural Networks With Low-rank Regularization GitHub Repo starsICLR2016
Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile ApplicationsICLR2016
Towards Convolutional Neural Networks Compression via Global Error ReconstructionIJCAI2016
Accelerating Convolutional Neural Networks for Mobile ApplicationsMM2016
Ultimate tensorization: compressing convolutional and FC layers alike GitHub Repo starsNIPSW2016
Speeding-up Convolutional Neural Networks Using Fine-tuned CP-DecompositionICLR2015
Speeding up Convolutional Neural Networks with Low Rank ExpansionsBMVC2014
Exploiting Linear Structure Within Convolutional Networks for Efficient EvaluationNIPS2014

RNN

📚 Surveys

📘 Miscellaneous

💻 Repositories

❓ Workshops

canonical-polyadic-decomposition
low-rank-factorization
model-compression
singular-value-decomposition
svd
tensor-decomposition
tensor-ring-decomposition
tensor-train
tucker-decomposition

vantienpham/Awesome-Tensor-Decomposition

😎 A curated list of tensor decomposition resources for model compression.

121

248 commits

updated Sep 20, 2026

See the code

README

Awesome Tensor Decomposition

A curated list of tensor decomposition resources for model compression.

📋 Research Papers

Transformer, LLM & more

TitleVenueYear
Understanding Calibration and Truncation Error Propagation in Training-Free Low-Rank Compression for LLMsCOLM2026
Surviving Resource-constraint Compression: Capability Retention under Tensor-train Decomposition for Sub-billion Parameter Language ModelsMachine Intelligence Research2026
Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression GitHub Repo starsICML2026
Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression GitHub Repo starsICML2026
Advancing SVD-based LLM Compression via Layer-Wise Error Model Search GitHub Repo starsICML2026
CARE: Covariance-Aware and Rank-Enhanced Decomposition for Enabling Multi-Head Latent Attention GitHub Repo starsICLR2026
Scalable Kronecker-Factored Fisher Approximation for Neural Network Parameter Sensitivity GitHub Repo starsICLR2026
LeSTD: LLM Compression via Learning-based Sparse Tensor DecompositionICLR2026
Large Language Model Compression with Global Rank and Sparsity OptimizationICLR2026
UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMsICLR2026
WSVD: Weighted Low-Rank Approximation for Fast and Efficient Execution of Low-Precision Vision-Language ModelsICLR2026
TD-MoE: Tensor Decomposition for MoE ModelsICLR2026
Efficient Resource-Constrained Training of Vision Transformers via Subspace OptimizationICLR2026
Theoretical guarantees for low-rank compression of deep neural networksApplied and Computational Harmonic Analysis2026
FLAT-LLM: Fine-grained Low-rank Activation Space Transformation for Large Language Model Compression GitHub Repo starsFindings of the Association for Computational Linguistics2026
QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Models GitHub Repo starsNeurIPS2025
MoE-SVD: Structured Mixture-of-Experts LLMs Compression via Singular Value DecompositionICML2025
LS-PRISM: A layer-selective pruning method via low-rank approximation and sparsification for efficient large language model compressionNeural Networks2025
Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device LearningICML2025
Canonical Rank Adaptation: An Efficient Fine-Tuning Strategy for Vision Transformers GitHub Repo starsICML2025
SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression GitHub Repo starsAAAI2025
SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression GitHub Repo starsNAACL2025
NdLinear Is All You Need for Representation Learning GitHub Repo starsArxiv2025
xKV: Cross-Layer SVD for KV-Cache Compression GitHub Repo starsArxiv2025
You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning GitHub Repo starsICLR2025
SliceGPT: Compress Large Language Models by Deleting Rows and Columns GitHub Repo starsICLR2025
Efficient Learning With Sine-Activated Low-rank MatricesICLR2025
Palu: Compression KV-Cache with Low-Rank Decomposition GitHub Repo starsICLR2025
Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives GitHub Repo starsICLR2025
MoDeGPT: Modular Decomposition for Large Language Model CompressionICLR2025
Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression GitHub Repo starsICLR2025
SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression GitHub Repo starsICLR2025
HASSLE-free: A unified Framework for Sparse plus Low-Rank Matrix Decomposition for LLMs GitHub Repo starsCPAL2025
FLAR-SVD: Fast and Latency-Aware Singular Value Decomposition for Model Compression GitHub Repo starsCVPRW2025
Breaking the Low-Rank Dilemma of Linear Attention GitHub Repo starsCVPR2025
FlexiGPT: Pruning and Extending Large Language Models with Low-Rank Weight SharingNAACL2025
TensorLLM: Tensorising Multi-Head Attention for Enhanced Reasoning and Compression in LLMsArxiv2025
Memory-Efficient Vision Transformers: An Activation-Aware Mixed-Rank Compression StrategyECCVW2024
Searching for Efficient Linear Layers over a Continuous Space of Structured Matrices GitHub Repo starsNeurIPS2024
DBA: Efficient Transformer With Dynamic Bilinear Low-Rank AttentionTNNLS2024
Tensor Product Attention Is All You Need GitHub Repo starsArxiv2025
EoRA: Training-free Compensation for Compressed LLM with Eigenspace Low-Rank ApproximationArxiv2024
Compressing Large Language Models using Low Rank and Low Precision Decomposition GitHub Repo starsNeurIPS2024
ESPACE: Dimensionality Reduction of Activations for Model Compression GitHub Repo starsNeurIPS2024
MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition GitHub Repo starsEMNLP 2024 Findings2024
Tensor Train Low-rank Approximation (TT-LoRA): Democratizing AI with Accelerated LLMsArxiv2024
Dual-grained Lightweight StrategyTPAMI2024
Adaptive Feature-based Low-Rank Compression of Large Language Models via Bayesian OptimizationEMNLP findings2024
Geometry is All You Need: A Unified Taxonomy of Matrix and Tensor Factorization for Compression of Generative Language ModelsArxiv2024
From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients GitHub Repo starsArxiv2024
The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction GitHub Repo starsICLR2024
ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models GitHub Repo starsArxiv2024
AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuning GitHub Repo starsEMNLP2024
Adaptive Rank Selections for Low-Rank Approximation of Language Models GitHub Repo starsNAACL2024
Dynamic Low-rank Estimation for Transformer-based Language ModelsNAACL Findings2024
PELA: Learning Parameter-Efficient Models with Low-Rank Approximation GitHub Repo starsCVPR2024
Adaptive Rank Selections for Low-Rank Approximation of Language ModelsACL2024
TRAWL: Tensor Reduced and Approximated Weights for Large Language ModelsArxiv2024
Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantization GitHub Repo starsISCA2024
Unified Low-rank Compression Framework for Click-through Rate Prediction GitHub Repo starsKDD2024 ADS2024
LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models GitHub Repo starsNAACL2024
CoMERA: Computing- and Memory-Efficient Training via Rank-Adaptive Tensor OptimizationArxiv2024
FLORA: Fine-grained Low-Rank Architecture Search for Vision Transformer GitHub Repo starsWACV2024
FLoRA: Low-Rank Core Space for N-dimension GitHub Repo starsArxiv2024
Enhancing GAN Performance Through Neural Architecture Search and Tensor Decomposition GitHub Repo starsICASSP2024
A Computational Study of Matrix Decomposition Methods for Compression of Pre-trained Transformers GitHub Repo starsPACLIC2024
Singularformer: Learning to Decompose Self-Attention to Linearize the Complexity of Transformer GitHub Repo starsIJCAI2023
LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation GitHub Repo starsICML2023
Singularformer: Learning to Decompose Self-Attention to Linearize the Complexity of Transformer GitHub Repo starsIJCAI2023
FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer GitHub Repo starsAAAI2023
LightFormer: Light-weight Transformer Using SVD-based Weight Transfer and Parameter SharingACL2023
Low-Rank Updates of pre-trained Weights for Multi-Task LearningACL2023
Compressing Transformers: Features Are Low-Rank, but Weights Are Not!AAAI2023
TT-MLP: Tensor Train Decomposition on Deep MLPsIEEE Access2023
Matrix Compression via Randomized Low Rank and Low Precision Factorization GitHub Repo starsNeurIPS2023
Strategies for Applying Low Rank Decomposition to Transformer-Based ModelsNeurIPS2022
Hypoformer: Hybrid Decomposition Transformer for Edge-friendly Neural Machine TranslationEMNLP2022
Kronecker Decomposition for GPT CompressionACL2022
Deeply Tensor Compressed Transformers for End-to-End Object DetectionAAAI2022
Language model compression with weighted low-rank factorization GitHub Repo starsICLR2022
DRONE: Data-aware Low-rank Compression for Large NLP ModelsNeurIPS2021
Linformer: Self-Attention with Linear Complexity GitHub Repo starsArxiv2020
Lightweight and Efficient End-to-End Speech Recognition Using Low-Rank Transformer GitHub Repo stars GitHub Repo starsICASSP2020
A Tensorized Transformer for Language Modeling GitHub Repo starsNeurIPS2019
GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model ShrinkingNIPS2018

CNN

TitleVenueYear
Basis Sharing and Coefficient Pruning for Efficient Neural Network CompressionNeural Networks2026
A Novel Approach to Lossless Convolutional Neural Network Compression via Progressive Knowledge Distillation-Incorporated Low-Rank CompressionNeural Networks2026
Deep Neural Network Compression via Data-Driven Low-Rank Singular Value DecompositionJournal of Artificial Intelligence Research2026
Adaptive Tucker Decomposition-based Progressive Model Compression for Convolutional Neural NetworksExpert Systems with Applications2026
Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networks GitHub Repo starsNeurIPS2025
Coupled Tensor Decomposition for Compact Network Representation GitHub Repo starsTNNLS2025
Reduced storage direct tensor ring decomposition for convolutional neural networks compression GitHub Repo starsNeural Networks2025
Co-Exploring Structured Sparsification and Low-Rank Tensor Decomposition for Compact DNNsTNNLS2025
Rope-net: deep convolutional neural network via robust principal component analysisMachine Learning2025
QuaDCNN: Quantized compression of deep CNN based on tensor-train decomposition with automatic rank determinationNeurocomputing2025
An overview of tensor and matrix decomposition methods, applied to deep neural networksEvolving Systems2025
FETTA: Flexible and Efficient Hardware Accelerator for Tensorized Neural Network TrainingArxiv2025
Fine-grained hierarchical singular value decomposition for convolutional neural networks compression and accelerationNeurocomputing2025
Unified Framework for Pre-trained Neural Network Compression via Decomposition and Optimized Rank SelectionECML PKDD2025
Low-Rank Compression for IMC ArraysArxiv2025
Enhancing GANs With MMD Neural Architecture Search, PMish Activation Function, and Adaptive Rank Decomposition GitHub Repo starsIEEE ACCESS2024
Quantization Aware Factorization for Deep Neural Network CompressionJAIR2024
J-Tucker: Joint Compression Scheme for Efficient Deployment of Multi-Task Deep Learning Models on Edge DevicesIEEE Network2024
3D-KCPNet: Efficient 3DCNNs based on tensor mapping theoryNeurocomputing2024
Fully Tensorized Lightweight ConvLSTM Neural Networks for Hyperspectral Image ClassificationTNNLS2024
TEC-CNN: Towards Efficient Compressing Convolutional Neural Nets with Low-rank Tensor DecompositionTOMM2024
Activation Map Compression through Tensor Decomposition for Deep Learning GitHub Repo starsNeurIPS2024
Geometry-aware training of factorized layers in tensor Tucker formatNeurIPS2024
Robustness of Tensor Decomposition-Based Neural Network CompressionICIP2024
Stable Low-Rank CP Decomposition for Compression of Convolutional Neural Networks Based on SensitivityApplied Sciences2024
How to Train Your Unstable Looped Tensor NetworkJSTSP2024
Learning Low-Rank Tensor Cores with Probabilistic l0-Regularized Rank Selection for Model Compression GitHub Repo starsIJCAI2024
Structure-Preserving Network Compression Via Low-Rank Induced Training Through Linear Layers Composition GitHub Repo starsTMLR2024
Compact Model Training by Low-Rank Projection With Energy Transfer GitHub Repo starsTNNLS2024
An Accuracy-Preserving Neural Network Compression via Tucker DecompositionIEEE Transactions on Sustainable Computing2024
Convolution Filter Compression via Sparse Linear Combinations of Quantized BasisTNNLS2024
Coarse-To-Fine Tensor Trains for Compact Visual Representations GitHub Repo starsICML2024
Position: Tensor Networks are a Valuable Asset for Green AIICML2024
Compression-aware Training of Neural Networks using Frank-Wolfe GitHub Repo starsArxiv2024
A Practical Approach for Employing Tensor Train Decomposition in Edge DevicesIJPP2024
Neural Network Compression Based on Tensor Ring DecompositionTNNLS2024
Enhanced network compression through tensor decompositions and pruning GitHub Repo starsTNNLS2024
Deep Convolutional Neural Network Compression Method: Tensor Ring Decomposition with Variational Bayesian ApproachNeural Processing Letters2024
Deep Learning Model Compression With Rank Reduction in Tensor DecompositionTNNLS2023
MARS: Masked Automatic Ranks Selection in Tensor Decompositions GitHub Repo starsAISTATS2023
Mixed-TD: Efficient Neural Network Accelerator with Layer-Specific Tensor Decomposition GitHub Repo starsFPL2023
SVD-NAS: Coupling Low-Rank Approximation and Neural Architecture Search GitHub Repo starsWACV2023
How Informative is the Approximation Error from Tensor Decomposition for Neural Network Compression?ICLR2023
Learning Low-Rank Representations for Model CompressionIJCNN2023
Tensor shape search for efficient compression of tensorized data and neural networksApplied Soft Computing2023
Compressing convolutional neural networks with hierarchical Tucker-2 decomposition GitHub Repo starsApplied Soft Computing2023
Tensor shape search for efficient compression of tensorized data and neural networksApplied Soft Computing2023
An effective low-rank compression with a joint rank selection followed by a compression-friendly trainingNeural Networks2023
Joint matrix decomposition for deep convolutional neural networks compression GitHub Repo starsNeurocomputing2023
Training Acceleration of Low-Rank Decomposed Networks using Sequential Freezing and Rank QuantizationArxiv2023
Knowledge Transfer via Decomposing Essential Information in Convolutional Neural Networks GitHub Repo starsTNNLS2022
Compression of Deep Neural Networks based on quantized tensor decomposition to implement on reconfigurable hardware platformsNeural Networks2022
Teacher–student knowledge distillation based on decomposed deep feature representation for intelligent mobile applicationsExpert Systems with Applications2022
HODEC: Towards Efficient High-Order DEcomposed Convolutional Neural NetworksCVPR2022
Towards Practical Control of Singular Values of Convolutional Layers GitHub Repo starsNeurIPS2022
Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations GitHub Repo starsNeurIPS2022
Realistic acceleration of neural networks with fine-grained tensor decompositionNeurocomputing2022
BATUDE: Budget-Aware Neural Network Compression Based on Tucker DecompositionAAAI2022
Convolutional Neural Network Compression through Generalized Kronecker Product Decomposition GitHub Repo starsAAAI2022
Towards Compact Neural Networks via End-to-End Training: A Bayesian Tensor Approach with Automatic Rank Determination GitHub Repo starsSIMODS2022
A novel compact design of convolutional layers with spatial transformation towards lower-rank representation for image classification GitHub Repo starsKnowledge-Based Systems2022
Deep neural network compression by Tucker decomposition with nonlinear responseKnowledge-Based Systems2022
Nested compression of convolutional neural networks with Tucker-2 decompositionIJCNN2022
PSM-nets: Compressing Neural Networks with Product of Sparse MatricesIJCNN2022
PARS: Proxy-Based Automatic Rank Selection for Neural Network Compression via Low-Rank Weight ApproximationMathematics2022
A Design Space Exploration Methodology for Enabling Tensor Train Decomposition in Edge DevicesSAMOS2022
Spatiotemporal Joint Filter Decomposition in 3D Convolutional Neural NetworksNeurIPS2021
Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition GitHub Repo starsNeurIPS2021
Deeply Shared Filter Bases for Parameter-Efficient Convolutional Neural Networks GitHub Repo starsNeurIPS2021
Decomposable-Net: Scalable Low-Rank Compression for Neural Networks GitHub Repo starsIJCAI2021
Tensor Regression Networks GitHub Repo starsJMLR2021
Parameter Efficient Dynamic Convolution via Tensor Decomposition GitHub Repo starsBMVC2021
Nonlinear tensor train format for deep neural network compressionNeural Networks2021
Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization FrameworkCVPR2021
Pufferfish: Communication-efficient Models At No Extra Cost GitHub Repo starsMLSys2021
Learning-based Tensor Decomposition with Adaptive Rank Penalty for CNNs CompressionMIPR2021
Deep Convolutional Neural Network Compression via Coupled Tensor DecompositionJSTSP2021
Tensor Reordering for CNN CompressionICASSP2021
Block-term tensor neural networksNeural Networks2020
Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural NetworkECCV2020
Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer GitHub Repo starsCVPR2020
Few Sample Knowledge Distillation for Efficient Network Compression GitHub Repo starsCVPR2020
Factorized Higher-Order CNNs with an Application to Spatio-Temporal Emotion EstimationCVPR2020
Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification GitHub Repo starsCVPRW2020
T-Basis: a Compact Representation for Neural Networks GitHub Repo starsICML2020
PENNI: Pruned Kernel Sharing for Efficient CNN Inference GitHub Repo starsICML2020
A Novel Rank Selection Scheme in Tensor Ring Decomposition Based on Reinforcement Learning for Deep Neural NetworksICASSP2020
Holistic CNN Compression via Low-Rank Decomposition with Knowledge Transfer GitHub Repo starsTPAMI2019
LTNN: A Layerwise Tensorized Compression of Multilayer Neural NetworkTNNLS2019
Einconv: Exploring Unexplored Tensor Network Decompositions for Convolutional Neural Networks GitHub Repo starsNeurIPS2019
Efficient Neural Network Compression GitHub Repo starsCVPR2019
ADA-Tucker: Compressing deep neural networks via adaptive dimension adjustment tucker decompositionNeural Networks2019
RotDCF: Decomposition of Convolutional Filters for Rotation-Equivariant Deep Networks GitHub Repo starsICLR2019
Learning Filter Basis for Convolutional Neural Network Compression GitHub Repo starsICCV2019
Automated Multi-Stage Compression of Neural Networks GitHub Repo starsICCVW2019
Compressing Deep Models using Multi Tensor Train DecompositionICCAIS2019
Compressing Fully Connected Layers using Kronecker Tensor DecompositionICCSNT2019
Adaptive Mixture of Low-Rank Factorizations for Compact Neural Modeling GitHub Repo starsOpenReview2019
Tensor-factorized neural networksTNNLS2018
Sharing Residual Units Through Collective Tensor Factorization To Improve Deep Neural NetworksIJCAI2018
Learning Compact Recurrent Neural Networks with Block-Term Tensor DecompositionCVPR2018
Wide Compression: Tensor Ring NetsCVPR2018
DCFNet: Deep Neural Network with Decomposed Convolutional Filters GitHub Repo starsICML2018
Self-supervised Knowledge Distillation Using Singular Value Decomposition GitHub Repo starsECCV2018
Extreme Network Compression via Filter Group ApproximationECCV2018
Network Decoupling: From Regular to Depthwise Separable Convolutions GitHub Repo starsBMVC2018
On Compressing Deep Models by Low Rank and Sparse DecompositionCVPR2017
Coordinating Filters for Faster Deep Neural Networks GitHub Repo starsICCV2017
Factorized Convolutional Neural NetworksICCVW2017
Tensor Regression Networks with various Low-Rank Tensor Approximations GitHub Repo starsArxiv2017
Accelerating Very Deep Convolutional Networks for Classification and DetectionTPAMI2016
Convolutional Neural Networks With Low-rank Regularization GitHub Repo starsICLR2016
Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile ApplicationsICLR2016
Towards Convolutional Neural Networks Compression via Global Error ReconstructionIJCAI2016
Accelerating Convolutional Neural Networks for Mobile ApplicationsMM2016
Ultimate tensorization: compressing convolutional and FC layers alike GitHub Repo starsNIPSW2016
Speeding-up Convolutional Neural Networks Using Fine-tuned CP-DecompositionICLR2015
Speeding up Convolutional Neural Networks with Low Rank ExpansionsBMVC2014
Exploiting Linear Structure Within Convolutional Networks for Efficient EvaluationNIPS2014

RNN

📚 Surveys

📘 Miscellaneous

💻 Repositories

❓ Workshops

canonical-polyadic-decomposition
low-rank-factorization
model-compression
singular-value-decomposition
svd
tensor-decomposition
tensor-ring-decomposition
tensor-train
tucker-decomposition

Significant stargazers

Peijie Dong

758 followers · starred Mar 2025

Thodoris Kouzelis

40 followers · starred Nov 2024