BAGEL-MoE-7B-GEN

This repository contains compressed variants of BAGEL-7B-MoT based on our paper:

Understanding and Harnessing Sparsity in Unified Multimodal Models [arXiv] · [GitHub]

We study sparsity in unified multimodal models that jointly handle image understanding and generation. Key findings:

• Understanding components tolerate substantial compression with minimal quality loss.
• Generation components are highly sensitive to pruning.
• We propose MoE Adaptation: partition the generation module into multiple experts and activate them sparsely, recovering performance while reducing active parameters.

The compressed models reduce the active parameters in the generation module by half while maintaining comparable or even improved GenEval scores.

GenEval Results

Installation

conda create -n efficient_ug python=3.10
conda activate efficient_ug
pip install -r requirements.txt

Evaluation

# GenEval evaluation
bash scripts/eval/bagel/run_geneval_wr.sh

Citation

@article{he2025sparsity,
  title   = {Understanding and Harnessing Sparsity in Unified Multimodal Models},
  author  = {He, Shwai and others},
  journal = {arXiv preprint arXiv:2512.02351},
  year    = {2025}
}