LoRA: Low-Rank Adaptation of Large Language Models
arXiv:2106.09685 · 52 repos reference this paper in their README
An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible. Using GPT-3 175B as an example -- deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive. We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. Compared to GPT-3 175B fine-tuned with Adam, LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times. LoRA performs on-par or better than fine-tuning in model quality on RoBERTa, DeBERTa, GPT-2, and GPT-3, despite having fewer trainable parameters, a higher training throughput, and, unlike adapters, no additional inference latency. We also provide an empirical investigation into rank-deficiency in language model adaptation, which sheds light on the efficacy of LoRA. We release a package that facilitates the integration of LoRA with PyTorch models and provide our implementations and model checkpoints for RoBERTa, DeBERTa, and GPT-2 at https://github.com/microsoft/LoRA.
webmachinelearning/prompt-api
777
·
·
💬 A proposal for a web API for prompting browser-provided language models
QwenLM/Qwen
21,710
·
·
The official repo of Qwen (通义千问) chat & pretrained large language model proposed by Alibaba Cloud.
webmachinelearning/writing-assistance-apis
172
·
·
✍️ A proposal for writing assistance web APIs: summarizer, writer, and rewriter
tatsu-lab/stanford_alpaca
30,243
·
·
Code and documentation to train Stanford's Alpaca models, and generate the data.
jbarnes850/deepseek-r1-finetune
60
·
·
A step by step guide to fine-tuning the DeepSeek R1 Distilled models on Apple Silicon machines.
Blealtan/RWKV-LM-LoRA
410
·
·
RWKV is a RNN with transformer-level LLM performance. It can be directly trained like a GPT…
Facico/Chinese-Vicuna
4,110
·
·
Chinese-Vicuna: A Chinese Instruction-following LLaMA-based Model —— 一个中文低资源的llama+lora方案，结构参考alpaca
zysNLP/quickllm
47
·
·
A repo for update and debug Mixtral-7x8B、MOE、ChatGLM3、LLaMa2、 BaChuan、Qwen an other LLM models…
microsoft/LoRA
13,787
·
·
Code for loralib, an implementation of "LoRA: Low-Rank Adaptation of Large Language Models"
h2oai/h2o-llmstudio
5,179
·
·
H2O LLM Studio - a framework and no-code GUI for fine-tuning LLMs. Documentation:…
linhduongtuan/BLOOM-LORA
183
·
·
Due to restriction of LLaMA, we try to reimplement BLOOM-LoRA (much less restricted BLOOM license…
Lightning-AI/lit-llama
6,084
·
·
Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and…
imsrgadich/llmtools
0
·
·
Fork from https://github.com/kuleshov-group/llmtools?tab=readme-ov-file#benchmark with my changes
boostcampaitech5/level3_nlp_finalproject-nlp-12
13
·
·
'내마리'는 나의 이야기에 귀를 기울임으로써 나에게 공감하고, 이야기의 맥락을 파악하고, 더 깊은 내용을 질문해주는 챗봇입니다.
