🎊 News

• [2026.05.12] Our DPA is accepted by ICML 2026!
• [2026.03.13] We open-source the code, weights and data of DPA!

📜 Brief Introduction

We introduce Deep Pre-Alignment (DPA), a simple yet effective architecture for improving multimodal alignment in VLMs.

Instead of directly projecting ViT features into the target language model, DPA employs a small VLM perceiver that pre-aligns visual representations with the perceiver LLM text space before they enter the target model. This allows the language model to focus on understanding and reasoning, rather than spending its early layers on modality alignment. Key highlights of DPA include:

• 💪 Better multimodal performance. DPA improves results across 8 multimodal benchmarks by +1.9.

• 🤝 Less language capability forgetting. DPA reduces language performance degradation by 32.9% across text benchmarks.

• ⚡ High computational efficiency. DPA introduces only minimal overhead compared to standard VLM architectures. It increases parameters by 1.17× and training cost by 1.14×, while retaining 94% inference throughput.

• 🔧 Plug-and-play design. DPA offers a seamless upgrade path for current VLM development, requiring only a modular replacement of the vision encoder.

📌Contents

• Dataset
• Install
• Model Weights
• Train
• Evaluation
  • Multimodal evaluation
  • Text evaluation
• Citation

Dataset

The PT dataset follows exactly LLaVA-Pretrain. We present the SFT dataset in this repository, which is the single-image part of the Stage 3 subset of MAmmoTH-VL-Instruct-12M. The dataset contains the image relative paths, conversations and meta infomation, while omits the original images.

Install

1. Clone this repository.

git clone https://github.com/THUMAI-Lab/Deep-Pre-Alignment.git
cd Deep-Pre-Alignment

2. Install package

bash setup_env.sh

Model Weights

Train

1. Prepare data

Prepare the LLaVA-Pretrain data and MAmmoTH-VL-Instruct-12M data. Then fill the paths to the dataset json/jsonl files and image directories in the training scripts in ms-swift/scripts.

2. Training

The scripts are designed for 32 A100 GPUs, you can adjust the batch size and gradient accumulation steps for your own hardware.

• Pretraining

Clone the DPA-4B-init model to the root of this repository and run the following commands for pre-training:

bash ms-swift/scripts/PT_DPA_4node.sh

• Fully Fine-tuning Fill the path to your pre-trained checkpoint in the SFT scripts in the following scripts, and run:

bash ms-swift/scripts/SFT_DPA_4node.sh

• LoRA Fill the path to your pre-trained checkpoint in the SFT scripts in the following scripts, and run:

bash ms-swift/scripts/SFT_DPA_4node_lora.sh

Evaluation

Multimodal Evaluation

1. Inference and evaluation

Please replace DEFAULT_BASE_URL and DEFAULT_API_KEY in Multimodal_eval/vlmevalkit/vlmeval/api/ori_gpt_client.py with a valid API URL and OpenAI api-key.

Note: The evaluation is based on gpt-4o except MMVet that uses gpt-4-turbo.

bash ms-swift/scripts/eval_DPA.sh "SEEDBench2_Plus MMVet MMStar MMMU_DEV_VAL MathVista_MINI MathVision OCRBench AI2D_TEST" /path/to/father/of/your/checkpoint/directory eval_step start_step poll_interval

The last 3 arguments means the script will start evaluating from start_step, evaluate every eval_step steps and check if there are new checkpoints every poll_interval seconds after finishing evaluation (0 for no checking). Inference should be run on 4 A100 GPUs.

2. Summarization

Run the following commands for summarization.

pip install openpyxl
python summarize_benchmarks_mm.py /path/to/father/of/your/checkpoint/directory

Text Evaluation

1. Inference and evaluation

Please prepare the evaluation data as written in opencompass/run_model.sh.

bash ms-swift/scripts/eval_DPA_text.sh /path/to/father/of/your/checkpoint/directory eval_step1 (eval_step2 ...)

Evaluation can be run on 2, 4 or 8 GPUs.

2. Summarization

Run the following commands for summarization.

python summarize_benchmarks_text.py /path/to/father/of/your/checkpoint/directory

Licenses

Usage and License Notices: The data, code, and checkpoint are intended and licensed for research use only. They are also restricted to uses that follow the license agreement of Qwen.

Acknowledgement

• ms-swift: The codebase we built upon.
• VLMEvalKit: The evaluation toolkit of multimodal benchmarks.
• opencompass: The evaluation toolkit of text benchmarks.\
• LLaVA: The pretrain data of DPA.
• MAmmoTH-VL: The fine-tuning data of DPA.

Citation

If you find our model/code/data/paper helpful, please consider cite our papers 📝 and star us ⭐️！

@article{
title={Deep Pre-Alignment for {VLM}s},
author={Tianyu Yu and Kechen Fang and Zihao Wan and Kaidong Zhang and Yicheng Zhang and Jun Song and Bo Zheng and Yuan Yao},
booktitle={Forty-third International Conference on Machine Learning},
year={2026},
}