Just helping myself keep track of LLM papers that I‘m reading, with an emphasis on inference and model compression.

Transformer Architectures

• Attention Is All You Need
• Fast Transformer Decoding: One Write-Head is All You Need - Multi-Query Attention
• Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
• Augmenting Self-attention with Persistent Memory (Meta 2019)
• MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers (Meta 2023)
• Hyena Hierarchy: Towards Larger Convolutional Language Models

Foundation Models

• LLaMA: Open and Efficient Foundation Language Models
• PaLM: Scaling Language Modeling with Pathways
• GPT-NeoX-20B: An Open-Source Autoregressive Language Model
• Language Models are Unsupervised Multitask Learners (OpenAI) - GPT-2
• BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
• OpenLLaMA: An Open Reproduction of LLaMA
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces

Position Encoding

• Self-Attention with Relative Position Representations
• RoFormer: Enhanced Transformer with Rotary Position Embedding - RoPE
• Transformer Language Models without Positional Encodings Still Learn Positional Information - NoPE
• Rectified Rotary Position Embeddings - ReRoPE

KV Cache

• H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models (Jun. 2023)
• vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Activation

• Searching for Activation Functions
• GLU Variants Improve Transformer
• PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU

Pruning

• Optimal Brain Damage (1990)
• Optimal Brain Surgeon (1993)
• Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning (Jan. 2023) - Introduces Optimal Brain Quantization based on the Optimal Brain Surgeon
• Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon
• SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
• A Simple and Effective Pruning Approach for Large Language Models - Introduces Wanda (pruning with Weights and Activations)

Quantization

• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale - Quantization with outlier handling. Might be solving the wrong problem - see "Quantizable Transformers" below.
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models - Another approach to quantization with outliers
• Up or Down? Adaptive Rounding for Post-Training Quantization (Qualcomm 2020) - Introduces AdaRound
• Understanding and Overcoming the Challenges of Efficient Transformer Quantization (Qualcomm 2021)
• QuIP: 2-Bit Quantization of Large Language Models With Guarantees (Cornell Jul. 2023) - Introduces incoherence processing
• SqueezeLLM: Dense-and-Sparse Quantization (Berkeley Jun. 2023)
• Intriguing Properties of Quantization at Scale (Cohere May 2023)
• Pruning vs Quantization: Which is Better? (Qualcomm Jul. 2023)

Normalization

• Root Mean Square Layer Normalization
• Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing - Introduces gated attention and argues that outliers are a consequence of normalization

Sparsity and rank compression

• Compressing Pre-trained Language Models by Decomposition - vanilla SVD composition to reduce matrix sizes
• Language model compression with weighted low-rank factorization - Fisher information-weighted SVD
• Numerical Optimizations for Weighted Low-rank Estimation on Language Model - Iterative implementation for the above
• Weighted Low-Rank Approximation (2003)
• Transformers learn through gradual rank increase
• Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models
• Scatterbrain: Unifying Sparse and Low-rank Attention Approximation
• LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation
• LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression
• KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillation
• TRP: Trained Rank Pruning for Efficient Deep Neural Networks - Introduces energy-pruning ratio

Fine-tuning

• LoRA: Low-Rank Adaptation of Large Language Models
• QLoRA: Efficient Finetuning of Quantized LLMs
• DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation - works over a range of ranks
• Full Parameter Fine-tuning for Large Language Models with Limited Resources

Sampling

• Mirostat: A Neural Text Decoding Algorithm that Directly Controls Perplexity
• Stay on topic with Classifier-Free Guidance

Scaling

• Efficiently Scaling Transformer Inference (Google Nov. 2022) - Pipeline and tensor parallelization for inference
• Megatron-LM (Nvidia Mar. 2020) - Intra-layer parallelism for training

Mixture of Experts

• Adaptive Mixtures of Local Experts (1991, remastered PDF)
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (Google 2017)
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (Google 2022)
• Go Wider Instead of Deeper

Watermarking

• A Watermark for Large Language Models

More

• Efficient Deep Learning Systems: Week 9, Compression
• The Transformer Family Version 2.0 (Lilian Weng)
• Large Transformer Model Inference Optimization (Lilian Weng)