Model Phase Transitions (MPTs)

Navigating Model Phase Transitions to Enable Extreme Lossless Compression

---

📰 News

[2026] Our work Model Phase Transitions in Large Language Model Compression has been accepted as a Perspective by
npj | Artificial Intelligence (Nature Portfolio).

This paper provides a unifying theoretical and empirical framework for understanding phase transitions in model compression, revealing how orthogonal redundancy sources jointly determine the limits of lossless compression in large language models.

📄 Paper: npj | Artificial Intelligence (Nature)
🔗 https://www.nature.com/articles/s44387-026-00072-8

---

🌐 Introduction

The increasing size and complexity of large language models (LLMs) present significant computational and memory challenges. To address these challenges, model compression techniques have been developed to reduce the resource demands of LLMs. However, compression techniques can exhibit Model Phase Transitions (MPT), where the model's performance remains stable until a critical threshold is reached, after which the performance degrades sharply. The key to achieving effective compression lies in understanding model redundancy, which provides buffers that allow for compression until this phase transition point is reached.

In our work, we introduce the concept of Model Phase Transition to fundamentally characterize performance degradation and lossless compression limits in large language models. Across more than thirty pruning, quantization, and low-rank decomposition techniques, we fit performance trajectories with a novel piecewise power-law–exponential curve that pinpoints each technique’s phase-transition point (PTP): structured pruning fails at 30–45% sparsity, unstructured pruning at 55–65%, low-rank decomposition at 17–30%, and quantization below 3-bit precision. Crucially, these redundancy sources are orthogonal: by staying within the joint safe region defined by their PTPs, we raise the prior single-dimension bottleneck, enabling lossless compression to 10% of original model size.

📬 Contact

If you find any errors or have suggestions, feel free to reach out: [email protected]

📖 Table of Contents

• 🌐 Introduction
• 📬 Contact
• 🔍 Core Concept: Model Redundancy & Phase Transitions
  • 🧱 Three Types of Model Redundancy
  • ⚡️ Model Phase Transition (MPT)
  • Major Findings on MPT (Model Phase Transition) Research
  • ⭐️ Major Findings on Combined Pruning and Quantization
• 📚 Papers
  • 💡 The Necessity of Low-Resource LLM Deployment
  • ✂️ Structured Pruning
  • 🧨 Unstructured Pruning
  • 📦 Quantization
  • 🧩 Low-Rank Decomposition
• 📊 Case Studies
  • Sensitivity to Pruning
    • Features
    • Experimental Evaluation of Structured Pruning Methods Results
    • Experimental Evaluation of Unstructured Pruning Methods Results
    • Comparison of Unstructured and Semi-structured Pruning Methods
  • Sensitivity to quantization
    • Features
  • Sensitivity to Low-Rank Decomposition
    • Features
  • ⭐️ Sensitivity to Combined Model Compression
  • 🥇 Horizontal comparisons across different compression strategies
• Datasets
• Main Models Used
• 📎 Citation

🔍 Core Concept: Model Redundancy & Phase Transitions

Fig. 1: This figure highlights three main types of redundancy: structural, numerical, and algebraic redundancy. Structural redundancy is managed through pruning, numerical redundancy through quantization, and algebraic redundancy through low-rank decomposition. These redundancies act as buffers, allowing for lossless model compression until the phase transition point is reached. The phase transition point remains stable when different types of compression methods are used together, enabling lossless compression of large models to about 10\% of their original size.

🧱 Three Types of Model Redundancy

Large Language Models (LLMs) exhibit inherent redundancies that enable compression:

⚡️ Model Phase Transition (MPT)

“The difference between a robustly-compressed model and a broken one is a single step past the PTP（Phase Transition Point）.”

• Definition Gradual degradation up to a critical compression ratio s₀; exponential collapse beyond. Our piece-wise power-law + exponential curve fits 30 + methods.
• Typical PTPs
  • Structured pruning: 30–45 % sparsity
  • Unstructured pruning: 55–65 % sparsity
  • Quantization: ≥ 3-bit precision to stay safe
  • Low-rank: ≥ 17–30 % sparsity

Major Findings on MPT (Model Phase Transition) Research

1. Structured Pruning Phase Transition

Fig. 2: This figure presents the perplexity (PPL) of several structured pruning methods across different sparsity ratios, including both experimental data and fitted curves. The stars indicate the turning points of the piecewise fitting curves, where the x-coordinate corresponds to the model’s phase transition point. As the sparsity ratio increases, the model's performance sharply degrades once the phase transition point is exceeded.

2. Unstructured Pruning Phase Transition

Fig. 3: This figure shows the perplexity (PPL) of several unstructured pruning methods across different sparsity ratios, including both experimental data and fitted curves. The stars indicate the turning points of the piecewise fitting curves, where the x-coordinate corresponds to the model’s phase transition point. The performance remains stable at lower sparsity ratios, but once the critical threshold is crossed, performance degradation becomes exponential.

3. Quantized Model Performance

Fig. 4: This figure investigates the performance of quantized models on WikiText2 across various parameter sizes (GB). The performance is measured in perplexity (PPL), and several model families (Qwen2.5, LLaMA-2, and Gemma-3) are compared at different quantization levels. The results highlight a phase transition point where compression leads to catastrophic performance degradation at 2-bit quantization, with larger models showing better robustness as the model size increases.

4. Low-Rank Decomposition Phase Transition

Fig. 5: This figure presents the perplexity (PPL) of several low-rank decomposition methods (ASVD and SVD-LLM) across varying sparsity ratios. The experimental data and fitted curves demonstrate a clear phase transition where performance sharply drops beyond a certain sparsity threshold. The turning points of the piecewise fitting curves are marked with stars, which indicate critical points in the model’s performance degradation.

Major Findings on Combined Pruning and Quantization

Combined Pruning and Quantization

Fig. 6: Panel a presents the results of combining GGUF quantization and Wanda pruning on the LLaMA2-7b model. The 3D surface plot (Panel a) illustrates how perplexity (PPL) varies across different pruning ratios and quantization levels. The figure shows how the performance degrades with increasing compression, with the surface becoming steeper beyond a certain threshold. In Panel b, the 2D contour projection highlights the best trade-off curve between model size and perplexity. The red line marks the minimal PPL achieved at equivalent compression ratios, while the orange curve shows the phase transition line (PTL), indicating the point beyond which model performance rapidly deteriorates.

📚 Papers

💡 The Necessity of Low-Resource LLM Deployment

• ⭐️ Nature Machine Intelligence
  Parameter-efficient fine-tuning of large-scale pre-trained language models

• ⭐️ Nature Communications
  Efficient GPT-4V level multimodal large language model for deployment on edge devices

✂️ Structured Pruning

• ⭐️ SliceGPT (ICLR'24)
  SliceGPT: Compress Large Language Models by Deleting Rows and Columns
  Code

• LLM-Pruner (NeurIPS'23)
  LLM-Pruner: On the Structural Pruning of Large Language Models
  Code

• SLEB (ICML'24)
  Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks
  Code

• FLAP (AAAI'24)
  Fluctuation-based Adaptive Structured Pruning for Large Language Models
  Code

• ShortGPT (arXiv)
  Layers in Large Language Models are More Redundant Than You Expect

• Sheared LLaMA (arXiv)
  Accelerating Language Model Pre-training via Structured Pruning
  Code

• Shortened LLaMA (ICLR'24 Workshop)
  Depth Pruning for LLMs with Comparison of Retraining Methods

• SRAD (ICLR'25)
  The Unreasonable Ineffectiveness of the Deeper Layers

• LoRAPrune (ACL'24 Findings)
  Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning
  Code

🧨 Unstructured Pruning

• ⭐️ Wanda (ICML'23)
  A Simple and Effective Pruning Approach for Large Language Models
  Code

• ⭐️ SparseGPT (ICML'23)
  Massive Language Models Can Be Accurately Pruned in One-Shot
  Code

• ⭐️ ADMM (arXiv)
  Fast and Effective Weight Update for Pruned Large Language Models
  Code

• OWL (ICML'24)
  Outlier Weighed Layerwise Sparsity (OWL)
  Code

• RIA (ICLR'24)
  Plug-and-Play: Efficient Post-training Pruning for LLMs
  Code

• DSnoT (ICLR'24)
  Training-Free Fine-tuning for Sparse LLMs
  Code

• Flash-LLM (VLDB'24)
  Cost-Effective and Efficient Inference with Unstructured Sparsity
  Code

• BESA (ICLR'24)
  Blockwise Parameter-Efficient Sparsity Allocation
  Code

📦 Quantization

• GPTQ (ICLR'23)
  Accurate Post-Training Quantization for Generative Pre-trained Transformers
  Code

• AWQ (MLSys'24)
  Activation-aware Weight Quantization for LLMs
  Code

• SmoothQuant (ICML'23)
  Accurate and Efficient Post-Training Quantization
  Code

• LLM.int8() (NeurIPS'22)
  8-bit Matrix Multiplication for Transformers at Scale
  Code

• QLoRA (NeurIPS'23)
  Efficient Finetuning of Quantized LLMs
  Code

• ⭐️ GGUF
  New quantized format for efficient LLM deployment
  Code

• ZeroQuant (NeurIPS'22)
  Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
  Code

🧩 Low-Rank Decomposition

• ASVD (arXiv)
  Activation-aware SVD for Compressing Large Language Models
  Code

• ⭐️ SVD-LLM (ICLR'25)
  Truncation-aware SVD for Large Language Model Compression
  Code

• LoSparse (ICML'23)
  Structured Compression based on Low-Rank and Sparse Approximation
  Code

• Lillama (NAACL'25)
  One-shot Compression via Local Feature Distillation
  Code

• MoDeGPT (ICLR'25)
  Modular Decomposition for Large Language Model Compression
  Code

📊 Case Studies

Sensitivity to Pruning

Fig. 7: This diagram illustrates structured pruning across model components, including neurons, attention heads, channels, sub-layers, and layers. The color intensity indicates the importance of each component: darker colors represent more critical parts, while lighter colors indicate less essential components, which are typically pruned first.

Features

🔍 Cardinal sparsity thresholds for pruning

Pruning methods across different categories have specific cardinal sparsity thresholds that greatly affect their performance.

Structured pruning typically exhibits a critical sparsity threshold around 50%. Beyond this point, performance starts to degrade significantly, often leading to the collapse of the model. While it accelerates inference, the trade-off is often not worth it as the model’s accuracy is heavily impacted.

Unstructured pruning and semi-structured pruning methods can achieve much higher sparsity levels, with a threshold around 80%.

🏁 Horizontal comparisons across structured pruning, unstructured pruning, and semi-structured pruning

In a horizontal comparison, at similar compression rates, unstructured pruning consistently outperforms semi-structured pruning, which in turn, outperforms structured pruning.

🛠️ Problems with Existing Pruning Methods

Structured pruning's limitations: While pruning techniques have shown promise, they come with inherent limitations and trade-offs: While the main advantage of structured pruning lies in inference speedup, as highlighted in several recent papers, practical observations reveal that the trade-off between accuracy and speed is not always favorable. When sparsity exceeds 50%, the model generally collapses, providing less than a 2x speedup.

Semi-structured pruning's weaknesses: Recently, semi-structured pruning methods are often discussed alongside unstructured pruning, but they fail to outperform unstructured pruning and don't offer clear advantages of their own. In practice, while they can achieve moderate compression, they struggle to maintain model performance compared to unstructured pruning methods. Their overall lack of competitiveness at comparable compression rates raises questions about their utility.

Challenges in unstructured pruning: Unstructured pruning methods can achieve high compression rates (typically over 70%), but the performance trade-off is significant. Despite promising results in theory, practical applications face difficulties with acceleration. Many studies that use methods like Wanda and SparseGPT as baselines focus on extreme compression rates (>70%) for performance comparisons. However, these rates tend to cause catastrophic performance degradation, often referred to as "model hemorrhaging." In contrast, Wanda and SparseGPT perform better at lower sparsity levels, where other improved models fall short. Future research may need to shift focus towards performance at lower compression rates, possibly exploring combinations of pruning with quantization techniques to achieve better compression and faster inference without sacrificing too much accuracy.

Experimental Evaluation of Structured Pruning Methods Results

Fig. 8: Performance comparison of different structured pruning methods applied to LLaMA2-7b, based on the WikiText2-PPL (Perplexity) metric, across varying sparsity levels. The chart highlights the impact of pruning on model performance, with lower PPL values indicating better performance.

Fig. 9: Perplexity performance of LLaMA-3-8B and OPT-6.7B under different pruning ratios (Pruned by Slicegpt).

Table 1: WikiText2-PPL results for structured pruning (w/o: without finetuning)

---

Experimental Evaluation of Unstructured Pruning Methods Results

Fig. 10: WikiText2 perplexity (PPL) trends for various structured pruning methods across sparsity levels ranging from 10% to 80%. The upper part of the figure shows the overall trend across the full sparsity spectrum, while the lower part zooms in on the 10%--60% range to highlight differences among pruning methods at moderate compression levels. Methods with "(ptb)" suffix indicate usage of the PTB calibration dataset in ablation studies. The results show that while many pruning methods suffer significant performance drops at higher sparsity, approaches like Wanda, SparseGPT, and ADMM consistently preserve model quality under lower sparsity, suggesting greater robustness and practical viability. Note that the magnitude pruning curve is omitted from the lower subplot, as it consistently shows the worst degradation and dominates the y-axis range in zoomed-in views.

---

Comparison of Unstructured and Semi-structured Pruning Methods

Fig. 11: Comparison of the performance of unstructured and structured pruning methods on WikiText2 PPL at 50% sparsity. The figure compares the perplexity (PPL) of various pruning methods for a model pruned to 50% sparsity. The methods include unstructured pruning (50%), and two types of structured pruning: 4:8 and 2:4, which indicate that 4 out of every 8 weights or 2 out of every 4 weights are pruned, respectively.

Table 2: WikiText2-PPL for different unstructured and semi-structured methods

Table 3: WikiText2-PPL for different unstructured and semi-structured methods calibrate on PTB dataset

Sensitivity to quantization

Fig. 12: GGUF progressive quantization with critical 3-bit threshold

Fig. 13: This figure compares symmetric (a) and asymmetric (b) quantization. In symmetric quantization, the intervals are evenly distributed around zero, while in asymmetric quantization, the intervals are non-symmetric, offering more flexibility for complex data distributions. The figure also shows the quantization and dequantization processes, with average loss values for both methods. Symmetric quantization typically results in higher average loss.

Features

🔍 Lossless quantization thresholds

Quantization exhibits a "safe compression zone" (critical 3-bit threshold), beyond which performance degrades nonlinearly—yet larger models retain superior performance under low-bit settings compared to smaller "full-scale" counterparts at the same memory footprint.

🏁 Horizontal comparison of different models and different quantification methods

We conducted weight-only quantization experiments on base models of approximately 7B parameters using GPTQ with 8-bit and 4-bit precision. The models evaluated include LLaMA-3-8B, Mistral-7B, OPT-6.7B, Orca2-7B, and BLOOM-7.1B. The results indicate that most models exhibit stable performance after both 8-bit and 4-bit quantization.

We evaluate several widely used Post-Training Quantization (PTQ) methods on LLaMA2-7b, using the WikiText2 dataset and a maximum input length of 2K. The evaluation focuses on PPL (Perplexity) to assess the model's accuracy retention after quantization. The results show that most quantization methods maintain good accuracy, with a maximum precision loss of only 5% at 4-bit quantization.

📉 Progressive quantization curves at full model scale

To systematically evaluate the impact of model quantization on inference performance (PPL, inference speed, ARC-Easy, ARC-Challenge, and MMLU), we conducted comprehensive experiments on multiple models quantized via the GGUF framework. These experiments covered progressive quantization from 1-bit to 16-bit precision, focusing on well-performing yet moderately sized Qwen-2.5 model families (0.5B、1.5B，3B，7B，14B，32B，72B). Furthermore, given the remarkable performance recently exhibited by DeepSeek-R1, we also incorporated several DeepSeek-R1-distilled variants of LLaMA-3.1 and Qwen-2.5.

Fig. 14: Perplexity of 5 models across quantization levels. (a) PTB. (b) Wikitext2.

Fig. 15: The performance of the Qwen-2.5 model under various GGUF quantization schemes across multiple datasets and scales. This figure presents four subplots stacked vertically: (Top) WikiText2 perplexity (PPL), (Second) ARC-Easy accuracy, (Third) ARC-Challenge accuracy, (Bottom) MMLU accuracy. Results are shown across multiple model scales (0.5B, 1.5B, 3B, 7B, 14B, 32B, and 72B). These subplots demonstrate the performance trends of the Qwen-2.5 model under different quantization settings, helping to illustrate the impact of GGUF quantization on both perplexity and accuracy across a range of tasks and model sizes.

Fig. 16: Impact of 2-bit quantization on PPL, ARC-Challenge accuracy, and MMLU accuracy across model scales for various model families (Gemma-3, LLaMA-2, Owner-2.5).

Table 4:Wikitext2-PPL results for various quantization methods with different bit configurations(2K).

Table 5: Comprehensive evaluation of quantized models including perplexities (PPL), inference speed, parameter sizes, and downstream task performance (ARC-Easy, ARC-Challenge, and MMLU).

Table 6: WikiText2-PPL and parameter sizes for LLaMA-2 and Gemma-3 model families.

---

Sensitivity to Low-Rank Decomposition

Features

🔍 Critical compression thresholds
In the domain of large language models, low-rank decomposition methods inherently offer limited compression ratios, as weight matrices in contemporary LLMs often exhibit near-full-rank characteristics. ASVD exhibits a phase transition at 17% rank reduction; beyond this point, PPL degrades exponentially. SVD-LLM’s phase transition occurs at 30%, indicating slightly higher compression tolerance but steeper collapse thereafter.

Table 7: Low-Rank Decomposition Results

Sensitivity to Combined Model Compression

This section examines synergistic effects of combined compression strategies. We employ LLaMA2-7B as the unified testbed across all experiments.

Table 8: WikiText2-PPL for ASVD + RTN Quantization (Low-rank decomposition + Quantization)

Table 9: WikiText2-PPL for SparseGPT + GPTQ Quantization (Unstructured pruning + Quantization)

Table 10: WikiText2-PPL for Wanda Pruning + GGUF Quantization (Unstructured pruning + Quantization)

---

Note: Each cell shows PPL (top) and relative model size (bottom) compared to the original FP16 model at 0% pruning.

🥇 Horizontal comparisons across different compression strategies

🏁 We compare various model compression methods, including both pruning and quantization techniques, under a 50% sparsity setting. The LLaMA2-7b model was tested on the WikiText2 dataset using Perplexity (PPL) as the performance metric. The results indicate that quantization outperforms pruning methods in terms of accuracy retention, with the ranking of performance being: Quantization > Unstructured Pruning > Semi-structured Pruning > Structured Pruning Low-Rank Decomposition.

Fig. 17: Comparative analysis of model compression approaches evaluating perplexity (PPL) on a log scale across two sparsity configurations. Top: 50% sparsity performance across four strategy types - quantization (PPL 5.4-5.8), unstructured pruning (PPL 5.9-14.9), semi-structured pruning (PPL 6.8-16.5), structured pruning (PPL 10.2-316.7), and low-rank decomposition (PPL 43.0-nan). Bottom: 4-bit quantization methods (PPL 5.5-5.8) versus 70% unstructured pruning techniques (PPL 12.1-52.4k). Lower values indicate better language modeling capability retention, demonstrating quantization's stability versus pruning's extreme PPL variance at high sparsity.

Datasets

Wikitext-2

• Description: Wikitext-2 is a dataset derived from English Wikipedia and is often used for language modeling tasks. It consists of a subset of Wikipedia articles, providing a rich source of information for training machine learning models.
• Link: Wikitext-2 on Hugging Face

PTB (Penn Treebank)

• Description: PTB is one of the most widely used datasets for language modeling and syntactic parsing. It contains a diverse range of English texts, including articles from the Wall Street Journal, and is commonly used to evaluate models for part-of-speech tagging, syntactic parsing, and language modeling.
• Link: PTB on Hugging Face

C4 (Colossal Clean Crawled Corpus)

• Description: C4 is a large-scale dataset built by scraping and cleaning web data. It is one of the most popular datasets for training models for various NLP tasks such as text generation, summarization, and question answering. The dataset is derived from web pages and contains a diverse range of information.
• Link: C4 on Hugging Face

MMLU (Massive Multitask Language Understanding)

• Description: MMLU is a benchmark dataset for evaluating a model’s ability to handle a wide range of tasks across multiple domains. It includes tasks related to science, math, humanities, and social science, and is used to evaluate the generalization capabilities of language models.
• Link: MMLU on Hugging Face

ARC-Challenge

• Description: ARC-Challenge is a dataset designed for evaluating models on challenging multiple-choice questions. It is part of the AI2 Reasoning Challenge and focuses on science questions requiring advanced reasoning.
• Link: ARC-Challenge on Hugging Face

ARC-Easy

• Description: ARC-Easy is another dataset from the AI2 Reasoning Challenge, designed with relatively simpler science questions compared to ARC-Challenge. It is meant for evaluating how well models handle straightforward reasoning tasks in the science domain.
• Link: ARC-Easy on Hugging Face

Main Models Used

Below are the primary models used in this project:

• LLaMA-2-70B: meta-llama/Llama-2-70b · Hugging Face
• LLaMA-2-13B: meta-llama/Llama-2-13b-chat-hf · Hugging Face
• LLaMA-2-7B: meta-llama/Llama-2-7b · Hugging Face
• LLaMA-3.1-8B-Instruct: meta-llama/Llama-3.1-8B-Instruct · Hugging Face
• Qwen2.5-72B-Instruct: Qwen/Qwen2.5-72B-Instruct · Hugging Face
• Qwen2.5-32B-Instruct: Qwen/Qwen2.5-32B-Instruct · Hugging Face
• Qwen2.5-14B-Instruct: Qwen/Qwen2.5-14B-Instruct · Hugging Face
• Qwen2.5-7B-Instruct: Qwen/Qwen2.5-7B-Instruct · Hugging Face
• Qwen2.5-3B-Instruct: Qwen/Qwen2.5-3B-Instruct · Hugging Face
• Qwen2.5-1.5B-Instruct: Qwen/Qwen2.5-1.5B-Instruct · Hugging Face
• Qwen2.5-0.5B-Instruct: Qwen/Qwen2.5-0.5B-Instruct · Hugging Face
• Gemma-3-27B-it: google/gemma-3-27b-it · Hugging Face
• Gemma-3-12B-it: google/gemma-3-12b-it · Hugging Face
• Gemma-3-4B-it: google/gemma-3-4b-it · Hugging Face
• Gemma-3-1B-it: google/gemma-3-1b-it · Hugging Face
• OPT-6.7B: facebook/opt-6.7b · Hugging Face
• Mistral-7B: mistralai/Mistral-7B-v0.1
• Orca-2-7B: microsoft/Orca-2-7b
• Bloom-7B: bigscience/bloom-7b1

---

📎 Citation

If you find this work useful, please consider citing:

@article{Ma2026PhaseTI,
  title={Phase transitions in large language model compression},
  author={Ziyang Ma and Zuchao Li and Lefei Zhang and Gui-Song Xia and Bo Du and Liangpei Zhang and Dacheng Tao},
  journal={npj Artificial Intelligence},
  year={2026},
  volume={2},
  url={https://api.semanticscholar.org/CorpusID:285307574}
}