Shadowgraph
TrendingAbout
Sign in

lliai/Awesome-Low-Rank-Adaptation

Awesome-Low-Rank-Adaptation

134

12 commits

updated Oct 13, 2024

See the code

README

A Comprehensive Survey on Low-Rank Adaptation

This repository provides a comprehensive survey of Low-Rank Adaptation (LoRA) methods and their applications. We welcome contributions to keep this list up-to-date. If you find this repository useful, please consider starring it.

Table of Contents

  1. LoRA Settings 1.1 Initialization 1.2 Hyperparameters 1.3 Optimization 1.4 Regularization
  2. Dynamic Rank
  3. LoRA Variants
  4. Other Low-rank Decomposition
  5. LoRA with Model Compressions 5.1 LoRA with Pruning 5.2 LoRA with Quantization 5.3 LoRA with NAS 5.4 Memory-efficient LoRA
  6. LoRA Extensions 6.1 Multiple LoRA 6.2 Mixture-of-Experts (MOE) LoRA 6.3 LoRA Merge
  7. LoRA applications 7.1 Visual Understanding 7.2 Visual Generation 7.3 Language Understanding 7.4 Multimodal learning 7.5 Other

1. LoRA Settings

YearTitleVenuePaperCode
2022LoRA: Low-Rank Adaptation of Large Language ModelsICLRLinkLink

1.1 Initialization

YearTitleVenuePaperCode
2024The Impact of Initialization on LoRA Finetuning Dynamics-Link-
2024ShareLoRA: Parameter Efficient and Robust Large Language Model Fine-tuning via Shared Low-Rank Adaptation-Link-
2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning-Link-
2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language ModelsICLRLinkLink
2024CorDA: Context-Oriented Decomposition Adaptation of Large Language ModelsarXivLinklink
2024SVFT: Parameter-Efficient Fine-Tuning with Singular VectorsarXivLinklink

1.2 Hyperparameters

YearTitleVenuePaperCode
2024LoRA+: Efficient Low Rank Adaptation of Large ModelsarXivLinkLink
2023The expressive power of low-rank adaptationICLRLinkLink

1.3 Optimization

YearTitleVenuePaperCode
2024Derivative-Free Optimization for Low-Rank Adaptation in Large Language ModelsarXivLinkLink
2024AFLoRA: Adaptive Freezing of Low Rank Adaptation in Parameter Efficient Fine-Tuning of Large ModelsarXivLink-
2023Bayesian Low-rank Adaptation for Large Language ModelsICLRLinkLink
2024A Study of Optimizations for Fine-tuning Large Language Models-Link-
2024Bayesian-LoRA: LoRA based Parameter Efficient Fine-Tuning using Optimal Quantization levels and Rank Values trough Differentiable Bayesian Gates-Link-
2024Understanding Linear Probing then Fine-tuning Language Models from NTK Perspective-Link-
2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models-Link-

1.4 Regularization

YearTitleVenuePaperCode
2024LoRA Meets Dropout under a Unified FrameworkarXivLink-
2024AdvLoRA: Adversarial Low-Rank Adaptation of Vision-Language ModelsarXivLink-
2024PeriodicLoRA: Breaking the Low-Rank Bottleneck in LoRA OptimizationarXivLink-
2024LoRA Dropout as a Sparsity Regularizer for Overfitting Control---
2024LoRA-drop: Efficient LoRA Parameter Pruning based on Output Evaluation-Link-

1.5 Sparse

YearTitleVenuePaperCode
2024Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMs-LinkLink
2024Sparse High Rank Adapters-Link-
2024SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining-Link-
2023Sparse Low-rank Adaptation of Pre-trained Language ModelsEMNLPLinkLink
2024SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs-Link-
2024MLAE: Masked LoRA Experts for Parameter-Efficient Fine-Tuning-Link-

1.6 Bayesian

YearTitleVenuePaperCode
2023Bayesian Low-rank Adaptation for Large Language ModelsICLRLinkLink
2024Bayesian-LoRA: LoRA based Parameter Efficient Fine-Tuning using Optimal Quantization levels and Rank Values trough Differentiable Bayesian Gates-Link-
2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models-Link-

1.7 Robust

YearTitleVenuePaperCode
2024ShareLoRA: Parameter Efficient and Robust Large Language Model Fine-tuning via Shared Low-Rank Adaptation-Link-
2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation-LinkLink

2. Dynamic Rank

YearTitleVenuePaperCode
2023Adaptive Budget Allocation for Parameter-Efficient Fine-TuningICLRLinkLink
2023DyLoRA: Parameter-Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank AdaptationEACLLinkLink
2024MoRA: High-Rank Updating for Parameter-Efficient Fine-TuningarXivLinkLink
2024BiLoRA: A Bi-level Optimization Framework for Overfitting-Resilient Low-Rank Adaptation of Large Pre-trained ModelsarXivLink-
2024Unlocking the Global Synergies in Low-Rank Adapters-Link-
2024ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Model-Link-

3. LoRA Variants

YearTitleVenuePaperCode
2023Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuningarXivLink-
2023VERA: VECTOR-BASED RANDOM MATRIX ADAPTATIONarXivLink-
2024DoRA: Weight-Decomposed Low-Rank AdaptationICMLLinkLink
2024FLoRA: Low-Rank Core Space for N-dimensionarXivLinkLink
2024Mixture-of-Subspaces in Low-Rank Adaptation-LinkLink
2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters-Link-
2024ReFT: Representation Finetuning for Language ModelsPreprintLinkLink
2024LaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional Adaptation-Link-
2024Structured Unrestricted-Rank Matrices for Parameter Efficient Fine-tuningPreprintLink
2024LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language ModelsNAACLLinkLink
2024Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models-LinkLink
2024Trans-LoRA: towards data-free Transferable Parameter Efficient Finetuning-Link-
2024VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks-Link-
2023Tied-LoRA: Enhancing parameter efficiency of LoRA with Weight Tying-Link-
2024Towards Modular LLMs by Building and Reusing a Library of LoRAs-Link-
2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning---
2024SIBO: A Simple Booster for Parameter-Efficient Fine-Tuning-Link-
2024Asymmetry in Low-Rank Adapters of Foundation Models-Link-
2024PROLORA: Partial Rotation Empowers More Parameter-Efficient LoRA-Link-
2024AFLoRA: Adaptive Freezing of Low Rank Adaptation in Parameter Efficient Fine-Tuning of Large Models-Link-
2023Increasing model capacity for free: A simple strategy for parameter efficient fine-tuningICLR--

4. Other Low-rank Decomposition

YearTitleVenuePaperCode
2024Parameter-Efficient Fine-Tuning with Discrete Fourier TransformICMLLinkLink
2024OLoRA: Orthonormal Low-Rank Adaptation of Large Language Models-Link-
2024Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection AdaptationarXivLinklink

5. LoRA with Model Compressions

5.1 LoRA with Pruning

YearTitleVenuePaperCode
2024RankAdaptor: Hierarchical Dynamic Low-Rank Adaptation for Structural Pruned LLMs-Link-
2024PRILoRA: Pruned and Rank-Increasing Low-Rank AdaptationEACLLink-
2023Pruning meets low-rank parameter-efficient fine-tuning-Link-

5.2 LoRA with Quantization

YearTitleVenuePaperCode
2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language ModelsICLRLinkLink
2024Low-Rank Quantization-Aware Training for LLMs-Link-
2024QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model TuningAAAI WorkshopLink-
2024LoQT: Low Rank Adapters for Quantized Training-Link-
2024One QuantLLM for ALL: Fine-tuning Quantized LLMs Once for Efficient Deployments-Link-
2023QLORA: Efficient Finetuning of Quantized LLMsNeurIPSLinkLink
2024Accurate LoRA-Finetuning Quantization of LLMs via Information RetentionICMLLinkLink

5.3 LoRA with NAS

YearTitleVenuePaperCode
2024LoNAS: Elastic Low-Rank Adapters for Efficient Large LanguageCOLINGLinkLink
2024Shears: Unstructured Sparsity with Neural Low-rank Adapter Search-Link-

5.4 Memory-efficient LoRA

YearTitleVenuePaperCode
2024Galore: Memory-efficient llm training by gradient low-rank projectionICMLLinkLink
2024Flora: Low-Rank Adapters Are Secretly Gradient CompressorsICMLLink-
2024BlockLLM: Memory-Efficient Adaptation of LLMs by Selecting and Optimizing the Right Coordinate BlocksPreprintLink

5.4 Knowledge Distillation LoRA

YearTitleVenuePaperCode
2024PC-LoRA: Low-Rank Adaptation for Progressive Model Compression with Knowledge Distillation-Link-

6. LoRA Extensions

6.1 Multiple LoRA

YearTitleVenuePaperCode
2024LoRA-Ensemble: Efficient Uncertainty Modelling for Self-attention Networks-Link-
2024MeteoRA: Multiple-tasks Embedded LoRA for Large Language Models-Link-
2024MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-TuningACLLink-
2023LoraHub: Efficient cross-task generalization via dynamic lora compositionICLRLink-
2024LoRA-Switch: Boosting the Efficiency of Dynamic LLM Adapters via System-Algorithm Co-design---

6.2 Mixture-of-Experts (MOE) LoRA

YearTitleVenuePaperCode
2023Loramoe: Revolutionizing mixture of experts for maintaining world knowledge in language model alignmentarXivLink-
2024MoLE: Mixture of LoRA ExpertsICLRLinkLink
2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts-Link-
2024AdaMoLE: Fine-Tuning Large Language Models with Adaptive Mixture of Low-Rank Adaptation Experts-Link-
2024Mixture of Experts Using Tensor Products-Link-

6.3 LoRA Merge

YearTitleVenuePaperCode

7. LoRA applications

7.1 Visual Understanding

YearTitleVenuePaperCode
2024Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything ModelICLRLink-
2024Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach-LinkLink
2024ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts-Link-
2023MeLo: Low-rank Adaptation is Better than Finetuning for Medical ImageLink

7.2 Visual Generation

YearTitleVenuePaperCode
2024ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts-Link-
2024MoE-FFD: Mixture of Experts for Generalized and Parameter-Efficient Face Forgery Detection-LinkLink
2024Mixture of Low-rank Experts for Transferable AI-Generated Image Detection-LinkLink
2024LoRA-Composer: Leveraging Low-Rank Adaptation for Multi-Concept Customization in Training-Free Diffusion Models-LinkLink
2024Low-Rank Few-Shot Adaptation of Vision-Language Models-Link-
2024FouRA: Fourier Low Rank Adaptation-Link-
2023Intrinsic LoRA: A Generalist Approach for Discovering Knowledge in Generative Models-LinkLink
2023Orthogonal Adaptation for Modular Customization of Diffusion ModelsPreprintLink
2023ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAsPreprintLinkLink
2023Cones: Concept Neurons in Diffusion Models for Customized GenerationICMLLink
2023Multi-Concept Customization of Text-to-Image DiffusionCVPRLink
2023Cones 2: Customizable Image Synthesis with Multiple SubjectsPreprintLinkLink
2024Block-wise LoRA: Revisiting Fine-grained LoRA for Effective Personalization and Stylization in Text-to-Image GenerationAAAILink
2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion ModelsNeurIPSLinkLink
2024SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated DataPreprintLinkLink
2024MACE: Mass Concept Erasure in Diffusion ModelsCVPRLink
2024DiffuseKronA: A Parameter Efficient Fine-tuning Method for Personalized Diffusion ModelPreprintLink
2024Multi-LoRA Composition for Image GenerationarXivLinkLink
2023Motion Style Transfer: Modular Low-Rank Adaptation for Deep Motion ForecastingPMLRLinkLink

7.3 Language Understanding

YearTitleVenuePaperCode
2023Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of RLHFarXivLinkLink

7.4 Multimodal learning

YearTitleVenuePaperCode
2024LaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional Adaptation-Link-
2024MoVA: Adapting Mixture of Vision Experts to Multimodal Context-LinkLink
2024AdvLoRA: Adversarial Low-Rank Adaptation of Vision-Language Models-Link

7.5 Federated Learning

YearTitleVenuePaperCode
2024Improving LoRA in Privacy-preserving Federated LearningICLRLink-
2024FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decomposition-Link-
2024FLoRA: Enhancing Vision-Language Models with Parameter-Efficient Federated Learning-Link-
2024FL-TAC: Enhanced Fine-Tuning in Federated Learning via Low-Rank, Task-Specific Adapter Clustering-Link-
2024DP-DyLoRA: Fine-Tuning Transformer-Based Models On-Device under Differentially Private Federated Learning using Dynamic Low-Rank Adaptation-Link-

7.6 Other

YearTitleVenuePaperCode
2023Low-Rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech RecognitionASRULink-
2024Low-Rank Adaptation of Time Series Foundational Models for Out-of-Domain Modality Forecasting-Link-
2023Continual Learning with Low Rank AdaptationNeurIPS WorkshopLink-
2024Zero-Shot Cross-Domain Dialogue State Tracking via Dual Low-Rank AdaptationACLLink-

Contributing

We welcome contributions to this survey. Please feel free to submit a pull request to add new papers or update existing information.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contributors

lliai

11 commits

suntea233

1 commits

Similar repos

Yuheng2000/Awesome-LoRA

Awesome Low-Rank Adaptation

62

ZJU-LLMs/Awesome-LoRAs

279

WenkeHuang/Awesome-MLLM-Tuning

Multimodal Large Language Model (MLLM) Tuning Survey: Keeping Yourself is Important in Downstream…

103

JustQJ/LLM_Materials

记录平时阅读的有关大模型的一些材料

2

AIoT-MLSys-Lab/Efficient-LLMs-Survey

[TMLR 2024] Efficient Large Language Models: A Survey

1,257

tiingweii-shii/Awesome-Resource-Efficient-LLM-Papers

a curated list of high-quality papers on resource-efficient LLMs 🌱

165

horseee/Awesome-Efficient-LLM

A curated list for Efficient Large Language Models

2,043

jacksonchen1998/LLaMA-Paper-List

Collection of papers using LLaMA as backbone model

50

lliai/Awesome-Low-Rank-Adaptation

Awesome-Low-Rank-Adaptation

134

12 commits

updated Oct 13, 2024

See the code

README

A Comprehensive Survey on Low-Rank Adaptation

This repository provides a comprehensive survey of Low-Rank Adaptation (LoRA) methods and their applications. We welcome contributions to keep this list up-to-date. If you find this repository useful, please consider starring it.

Table of Contents

  1. LoRA Settings 1.1 Initialization 1.2 Hyperparameters 1.3 Optimization 1.4 Regularization
  2. Dynamic Rank
  3. LoRA Variants
  4. Other Low-rank Decomposition
  5. LoRA with Model Compressions 5.1 LoRA with Pruning 5.2 LoRA with Quantization 5.3 LoRA with NAS 5.4 Memory-efficient LoRA
  6. LoRA Extensions 6.1 Multiple LoRA 6.2 Mixture-of-Experts (MOE) LoRA 6.3 LoRA Merge
  7. LoRA applications 7.1 Visual Understanding 7.2 Visual Generation 7.3 Language Understanding 7.4 Multimodal learning 7.5 Other

1. LoRA Settings

YearTitleVenuePaperCode
2022LoRA: Low-Rank Adaptation of Large Language ModelsICLRLinkLink

1.1 Initialization

YearTitleVenuePaperCode
2024The Impact of Initialization on LoRA Finetuning Dynamics-Link-
2024ShareLoRA: Parameter Efficient and Robust Large Language Model Fine-tuning via Shared Low-Rank Adaptation-Link-
2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning-Link-
2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language ModelsICLRLinkLink
2024CorDA: Context-Oriented Decomposition Adaptation of Large Language ModelsarXivLinklink
2024SVFT: Parameter-Efficient Fine-Tuning with Singular VectorsarXivLinklink

1.2 Hyperparameters

YearTitleVenuePaperCode
2024LoRA+: Efficient Low Rank Adaptation of Large ModelsarXivLinkLink
2023The expressive power of low-rank adaptationICLRLinkLink

1.3 Optimization

YearTitleVenuePaperCode
2024Derivative-Free Optimization for Low-Rank Adaptation in Large Language ModelsarXivLinkLink
2024AFLoRA: Adaptive Freezing of Low Rank Adaptation in Parameter Efficient Fine-Tuning of Large ModelsarXivLink-
2023Bayesian Low-rank Adaptation for Large Language ModelsICLRLinkLink
2024A Study of Optimizations for Fine-tuning Large Language Models-Link-
2024Bayesian-LoRA: LoRA based Parameter Efficient Fine-Tuning using Optimal Quantization levels and Rank Values trough Differentiable Bayesian Gates-Link-
2024Understanding Linear Probing then Fine-tuning Language Models from NTK Perspective-Link-
2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models-Link-

1.4 Regularization

YearTitleVenuePaperCode
2024LoRA Meets Dropout under a Unified FrameworkarXivLink-
2024AdvLoRA: Adversarial Low-Rank Adaptation of Vision-Language ModelsarXivLink-
2024PeriodicLoRA: Breaking the Low-Rank Bottleneck in LoRA OptimizationarXivLink-
2024LoRA Dropout as a Sparsity Regularizer for Overfitting Control---
2024LoRA-drop: Efficient LoRA Parameter Pruning based on Output Evaluation-Link-

1.5 Sparse

YearTitleVenuePaperCode
2024Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMs-LinkLink
2024Sparse High Rank Adapters-Link-
2024SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining-Link-
2023Sparse Low-rank Adaptation of Pre-trained Language ModelsEMNLPLinkLink
2024SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs-Link-
2024MLAE: Masked LoRA Experts for Parameter-Efficient Fine-Tuning-Link-

1.6 Bayesian

YearTitleVenuePaperCode
2023Bayesian Low-rank Adaptation for Large Language ModelsICLRLinkLink
2024Bayesian-LoRA: LoRA based Parameter Efficient Fine-Tuning using Optimal Quantization levels and Rank Values trough Differentiable Bayesian Gates-Link-
2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models-Link-

1.7 Robust

YearTitleVenuePaperCode
2024ShareLoRA: Parameter Efficient and Robust Large Language Model Fine-tuning via Shared Low-Rank Adaptation-Link-
2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation-LinkLink

2. Dynamic Rank

YearTitleVenuePaperCode
2023Adaptive Budget Allocation for Parameter-Efficient Fine-TuningICLRLinkLink
2023DyLoRA: Parameter-Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank AdaptationEACLLinkLink
2024MoRA: High-Rank Updating for Parameter-Efficient Fine-TuningarXivLinkLink
2024BiLoRA: A Bi-level Optimization Framework for Overfitting-Resilient Low-Rank Adaptation of Large Pre-trained ModelsarXivLink-
2024Unlocking the Global Synergies in Low-Rank Adapters-Link-
2024ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Model-Link-

3. LoRA Variants

YearTitleVenuePaperCode
2023Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuningarXivLink-
2023VERA: VECTOR-BASED RANDOM MATRIX ADAPTATIONarXivLink-
2024DoRA: Weight-Decomposed Low-Rank AdaptationICMLLinkLink
2024FLoRA: Low-Rank Core Space for N-dimensionarXivLinkLink
2024Mixture-of-Subspaces in Low-Rank Adaptation-LinkLink
2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters-Link-
2024ReFT: Representation Finetuning for Language ModelsPreprintLinkLink
2024LaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional Adaptation-Link-
2024Structured Unrestricted-Rank Matrices for Parameter Efficient Fine-tuningPreprintLink
2024LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language ModelsNAACLLinkLink
2024Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models-LinkLink
2024Trans-LoRA: towards data-free Transferable Parameter Efficient Finetuning-Link-
2024VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks-Link-
2023Tied-LoRA: Enhancing parameter efficiency of LoRA with Weight Tying-Link-
2024Towards Modular LLMs by Building and Reusing a Library of LoRAs-Link-
2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning---
2024SIBO: A Simple Booster for Parameter-Efficient Fine-Tuning-Link-
2024Asymmetry in Low-Rank Adapters of Foundation Models-Link-
2024PROLORA: Partial Rotation Empowers More Parameter-Efficient LoRA-Link-
2024AFLoRA: Adaptive Freezing of Low Rank Adaptation in Parameter Efficient Fine-Tuning of Large Models-Link-
2023Increasing model capacity for free: A simple strategy for parameter efficient fine-tuningICLR--

4. Other Low-rank Decomposition

YearTitleVenuePaperCode
2024Parameter-Efficient Fine-Tuning with Discrete Fourier TransformICMLLinkLink
2024OLoRA: Orthonormal Low-Rank Adaptation of Large Language Models-Link-
2024Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection AdaptationarXivLinklink

5. LoRA with Model Compressions

5.1 LoRA with Pruning

YearTitleVenuePaperCode
2024RankAdaptor: Hierarchical Dynamic Low-Rank Adaptation for Structural Pruned LLMs-Link-
2024PRILoRA: Pruned and Rank-Increasing Low-Rank AdaptationEACLLink-
2023Pruning meets low-rank parameter-efficient fine-tuning-Link-

5.2 LoRA with Quantization

YearTitleVenuePaperCode
2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language ModelsICLRLinkLink
2024Low-Rank Quantization-Aware Training for LLMs-Link-
2024QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model TuningAAAI WorkshopLink-
2024LoQT: Low Rank Adapters for Quantized Training-Link-
2024One QuantLLM for ALL: Fine-tuning Quantized LLMs Once for Efficient Deployments-Link-
2023QLORA: Efficient Finetuning of Quantized LLMsNeurIPSLinkLink
2024Accurate LoRA-Finetuning Quantization of LLMs via Information RetentionICMLLinkLink

5.3 LoRA with NAS

YearTitleVenuePaperCode
2024LoNAS: Elastic Low-Rank Adapters for Efficient Large LanguageCOLINGLinkLink
2024Shears: Unstructured Sparsity with Neural Low-rank Adapter Search-Link-

5.4 Memory-efficient LoRA

YearTitleVenuePaperCode
2024Galore: Memory-efficient llm training by gradient low-rank projectionICMLLinkLink
2024Flora: Low-Rank Adapters Are Secretly Gradient CompressorsICMLLink-
2024BlockLLM: Memory-Efficient Adaptation of LLMs by Selecting and Optimizing the Right Coordinate BlocksPreprintLink

5.4 Knowledge Distillation LoRA

YearTitleVenuePaperCode
2024PC-LoRA: Low-Rank Adaptation for Progressive Model Compression with Knowledge Distillation-Link-

6. LoRA Extensions

6.1 Multiple LoRA

YearTitleVenuePaperCode
2024LoRA-Ensemble: Efficient Uncertainty Modelling for Self-attention Networks-Link-
2024MeteoRA: Multiple-tasks Embedded LoRA for Large Language Models-Link-
2024MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-TuningACLLink-
2023LoraHub: Efficient cross-task generalization via dynamic lora compositionICLRLink-
2024LoRA-Switch: Boosting the Efficiency of Dynamic LLM Adapters via System-Algorithm Co-design---

6.2 Mixture-of-Experts (MOE) LoRA

YearTitleVenuePaperCode
2023Loramoe: Revolutionizing mixture of experts for maintaining world knowledge in language model alignmentarXivLink-
2024MoLE: Mixture of LoRA ExpertsICLRLinkLink
2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts-Link-
2024AdaMoLE: Fine-Tuning Large Language Models with Adaptive Mixture of Low-Rank Adaptation Experts-Link-
2024Mixture of Experts Using Tensor Products-Link-

6.3 LoRA Merge

YearTitleVenuePaperCode

7. LoRA applications

7.1 Visual Understanding

YearTitleVenuePaperCode
2024Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything ModelICLRLink-
2024Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach-LinkLink
2024ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts-Link-
2023MeLo: Low-rank Adaptation is Better than Finetuning for Medical ImageLink

7.2 Visual Generation

YearTitleVenuePaperCode
2024ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts-Link-
2024MoE-FFD: Mixture of Experts for Generalized and Parameter-Efficient Face Forgery Detection-LinkLink
2024Mixture of Low-rank Experts for Transferable AI-Generated Image Detection-LinkLink
2024LoRA-Composer: Leveraging Low-Rank Adaptation for Multi-Concept Customization in Training-Free Diffusion Models-LinkLink
2024Low-Rank Few-Shot Adaptation of Vision-Language Models-Link-
2024FouRA: Fourier Low Rank Adaptation-Link-
2023Intrinsic LoRA: A Generalist Approach for Discovering Knowledge in Generative Models-LinkLink
2023Orthogonal Adaptation for Modular Customization of Diffusion ModelsPreprintLink
2023ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAsPreprintLinkLink
2023Cones: Concept Neurons in Diffusion Models for Customized GenerationICMLLink
2023Multi-Concept Customization of Text-to-Image DiffusionCVPRLink
2023Cones 2: Customizable Image Synthesis with Multiple SubjectsPreprintLinkLink
2024Block-wise LoRA: Revisiting Fine-grained LoRA for Effective Personalization and Stylization in Text-to-Image GenerationAAAILink
2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion ModelsNeurIPSLinkLink
2024SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated DataPreprintLinkLink
2024MACE: Mass Concept Erasure in Diffusion ModelsCVPRLink
2024DiffuseKronA: A Parameter Efficient Fine-tuning Method for Personalized Diffusion ModelPreprintLink
2024Multi-LoRA Composition for Image GenerationarXivLinkLink
2023Motion Style Transfer: Modular Low-Rank Adaptation for Deep Motion ForecastingPMLRLinkLink

7.3 Language Understanding

YearTitleVenuePaperCode
2023Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of RLHFarXivLinkLink

7.4 Multimodal learning

YearTitleVenuePaperCode
2024LaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional Adaptation-Link-
2024MoVA: Adapting Mixture of Vision Experts to Multimodal Context-LinkLink
2024AdvLoRA: Adversarial Low-Rank Adaptation of Vision-Language Models-Link

7.5 Federated Learning

YearTitleVenuePaperCode
2024Improving LoRA in Privacy-preserving Federated LearningICLRLink-
2024FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decomposition-Link-
2024FLoRA: Enhancing Vision-Language Models with Parameter-Efficient Federated Learning-Link-
2024FL-TAC: Enhanced Fine-Tuning in Federated Learning via Low-Rank, Task-Specific Adapter Clustering-Link-
2024DP-DyLoRA: Fine-Tuning Transformer-Based Models On-Device under Differentially Private Federated Learning using Dynamic Low-Rank Adaptation-Link-

7.6 Other

YearTitleVenuePaperCode
2023Low-Rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech RecognitionASRULink-
2024Low-Rank Adaptation of Time Series Foundational Models for Out-of-Domain Modality Forecasting-Link-
2023Continual Learning with Low Rank AdaptationNeurIPS WorkshopLink-
2024Zero-Shot Cross-Domain Dialogue State Tracking via Dual Low-Rank AdaptationACLLink-

Contributing

We welcome contributions to this survey. Please feel free to submit a pull request to add new papers or update existing information.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Similar repos

Yuheng2000/Awesome-LoRA

Awesome Low-Rank Adaptation

62

ZJU-LLMs/Awesome-LoRAs

279

WenkeHuang/Awesome-MLLM-Tuning

Multimodal Large Language Model (MLLM) Tuning Survey: Keeping Yourself is Important in Downstream…

103

JustQJ/LLM_Materials

记录平时阅读的有关大模型的一些材料

2

AIoT-MLSys-Lab/Efficient-LLMs-Survey

[TMLR 2024] Efficient Large Language Models: A Survey

1,257

tiingweii-shii/Awesome-Resource-Efficient-LLM-Papers

a curated list of high-quality papers on resource-efficient LLMs 🌱

165

horseee/Awesome-Efficient-LLM

A curated list for Efficient Large Language Models

2,043

jacksonchen1998/LLaMA-Paper-List

Collection of papers using LLaMA as backbone model

50

Contributors

lliai

11 commits

suntea233

1 commits