Fully Open Framework for Democratized Multimodal Training

---

---

NEWS

• 2025-12-11: Released RL recipe for LLaVA-OneVision-1.5 with code, data, and model.
• 2025-09-30: Released the Offline Data Packing Guide.
• 2025-09-30: Released the LLaVA-OneVision-1.5 Technical Report.

Contents

• Introduction
• Models
• Datasets
• Results
• Quick Start with Hugging Face
• Evaluation
• Quick Start For Training
• Unbalanced Pipeline Splitting
• Fully Reproducing Guide
• Citation
• Acknowledgement

Evaluation

# pip install git+https://github.com/EvolvingLMMs-Lab/lmms-eval.git  

accelerate launch --num_processes=8 --main_process_port 12399 -m lmms_eval \
    --model=llava_onevision1_5 \
    --model_args=pretrained=lmms-lab/LLaVA-OneVision-1.5-8B-Instruct,attn_implementation=flash_attention_2,max_pixels=3240000 \
    --tasks=mmmu_val,mmmu_pro_standard,mmbench_en_test,mmerealworld,mmerealworld_cn,ai2d,ai2d_no_mask,vstar_bench,chartqa,charxiv,docvqa_test,mathvista_testmini,mmstar,scienceqa \
    --batch_size=1

Quick Start Guide

1.🐳 Docker (Recommended)

We strongly recommend using the docker environment for a seamless experience. The following instructions are tailored for the A100 80GB GPU environment.

# Clone repository
git clone https://github.com/anxiangsir/LLaVA-OneVision-2.git
cd LLaVA-OneVision-2

docker build -t llava_megatron:25.12 .

# Run container with -w to set working directory directly to the mounted volume
docker run -it --gpus all \
    --ipc host --net host --privileged --cap-add IPC_LOCK \
    --ulimit memlock=-1 --ulimit stack=67108864 --rm \
    -v $(pwd):/workspace/LLaVA-OneVision-2 \
    -w /workspace/LLaVA-OneVision-2 \
    --name "llava_megatron_container" \
    llava_megatron:25.12 /bin/bash

2. Checkpoint and Format Conversion

You have two options to get started with LLaVA-OneVision-1.5-stage-0:

Option 2: Merge initial weights yourself

Alternatively, you can merge the initial weights from the original ViT and LLM:

python ds/merge_model.py \
--vit_path DeepGlint-AI/rice-vit-large-patch14-560 \
--llm_path Qwen/Qwen3-4B-Instruct-2507 \
--output LLaVA-OneVision-1.5-4B-stage0

Note: When merging weights, the adapter component will be initialized with default values.

Convert the model from Hugging Face format to Megatron format:

AIAK_TRAINING_PATH=/workspace/LLaVA-OneVision-2 bash examples/llava_onevision1_5/convert/convert_4b_hf_to_mcore.sh \
LLaVA-OneVision-1.5-4B-stage0 \
LLaVA-OneVision-1.5-4B-stage0_mcore_tp1_pp1 \
1 1

3. Stage 1 Alignment-Training

Download LLaVA from LLaVA-558K-Webdataset.

# ============================================================
# Required environment variables:
#   AIAK_TRAINING_PATH  Root directory of the AIAK-Training-LLM project
#   DATA_PATH           Directory with WebDataset shards (.tar) for pretraining
#   TOKENIZER_PATH      Hugging Face tokenizer directory
#   CHECKPOINT_PATH     Megatron-formatted checkpoint directory (e.g., mcore TP1/PP1)
#   SAVE_CKPT_PATH      Output directory for saving training checkpoints
AIAK_TRAINING_PATH=/workspace/LLaVA-OneVision-2 \
DATA_PATH=LLaVA-558K-Webdataset \
TOKENIZER_PATH=LLaVA-OneVision-1.5-4B-stage0 \
CHECKPOINT_PATH=LLaVA-OneVision-1.5-4B-stage0_mcore_tp1_pp1 \
bash examples/llava_onevision1_5/quick_start/stage_1_alignment_llava_ov_4b.sh

4. Stage 1.5 Mid-Training

Download our lightweight packed subset from LLaVA-OneVision-1.5-Mid-Training-Quick-Start-3M-Webdataset.

# ============================================================
# Convert model to release format
bash examples/llava_onevision1_5/convert/convert_4b_mcore_to_release.sh \
stage_1_alignment_llava_ov_4b/iter_0002500/ \
stage_1_alignment_llava_ov_4b_release 1 1
# ============================================================
# Launch
AIAK_TRAINING_PATH=/workspace/LLaVA-OneVision-2 \
DATA_PATH=LLaVA-OneVision-1.5-Mid-Training-Webdataset-Quick-Start-3M \
TOKENIZER_PATH=LLaVA-OneVision-1.5-4B-stage0 \
CHECKPOINT_PATH=stage_1_alignment_llava_ov_4b_release \
bash examples/llava_onevision1_5/quick_start/stage_1.5_mid_training_llava_ov_4b.sh

5. Stage 2 Instruct-Training

Download LLaVA-NeXT-780k-webdataset at LLaVA-NeXT-780K Dataset.

# ============================================================
# Convert model to release format
bash examples/llava_onevision1_5/convert/convert_4b_mcore_to_release.sh \
stage_1.5_mid_training_llava_ov_4b/iter_0020000/ \
stage_1.5_mid_training_llava_ov_4b_release 1 1
# ============================================================
# # Launch
AIAK_TRAINING_PATH=/workspace/LLaVA-OneVision-2 \
DATA_PATH=LLaVA-NeXT-780k-Webdataset \
TOKENIZER_PATH=LLaVA-OneVision-1.5-4B-stage0 \
CHECKPOINT_PATH=stage_1.5_mid_training_llava_ov_4b_release \
bash examples/llava_onevision1_5/quick_start/stage_2_instruct_llava_ov_4b.sh

6. Convert mcore to Hugging Face

AIAK_TRAINING_PATH=/workspace/LLaVA-OneVision-2 \
bash examples/llava_onevision1_5/convert/convert_4b_mcore_to_hf.sh \
stage_2_instruct_llava_ov_4b/iter_0003500 \
LLaVA-OneVision-1.5-4B-3M-Mid-Training-780K-Instruct \
1 1
# Copy non-model files (e.g., tokenizer config) to the new directory
find LLaVA-OneVision-1.5-4B-stage0/ -type f -not -iname '*safetensors*' -exec cp {}  LLaVA-OneVision-1.5-4B-3M-Mid-Training-780K-Instruct/ ';'

7. Evaluation

# pip install git+https://github.com/EvolvingLMMs-Lab/lmms-eval.git
CUDA_VISIBLE_DEVICES=4,5,6,7 accelerate launch \
--num_processes=4 --main_process_port 12399 -m lmms_eval --model=llava_onevision1_5 --batch_size=1 --tasks=mme \
--model_args=pretrained=/workspace/LLaVA-OneVision-2/LLaVA-OneVision-1.5-4B-3M-Mid-Training-780K-Instruct,max_pixels=3240000

Unbalanced Pipeline Splitting (不均衡流水线切分)

Background

LLaVA-OneVision-2 is a multimodal model that consists of three components placed across pipeline stages:

• Stage 0 – ViT encoder (24 transformer layers, ~300 M parameters) plus optionally a few LLM decoder layers
• Stage 1 … N-1 – the remaining LLM decoder layers

Because the ViT encoder adds significant computation to stage 0, a naive uniform layer split would make stage 0 much heavier than the others, creating pipeline bubbles and lowering GPU utilisation.

Custom / unbalanced pipeline splitting lets you specify exactly how many LLM layers each pipeline stage holds, so you can compensate for the extra ViT load on stage 0 by giving it fewer (or zero) LLM layers.

Parameters

Note on --custom-pipeline-recompute-layers stage 0: Because stage 0 always hosts the ViT encoder (24 layers), the recompute budget for stage 0 applies to those ViT layers. For example, setting 12 for stage 0 means half of the ViT's 24 layers are recomputed. Stages 1+ cover the LLM layers assigned to that stage.

Recommended Configurations

2B model – 28 LLM layers, 24-layer ViT on stage 0

4B model – 36 LLM layers, 24-layer ViT on stage 0

8B model – 36 LLM layers, 24-layer ViT on stage 0

Training Script Example (4B, TP=1 PP=2)

The key training arguments for an unbalanced PP=2 run on the 4B model:

TRAINING_ARGS=(
    ...
    # Unbalanced layer split: stage-0 holds ViT (24 layers) + 12 LLM layers,
    # stage-1 holds the remaining 24 LLM layers.
    --custom-pipeline-layers 12,24

    # Per-stage activation recompute budget:
    #   stage 0 → 12 out of 24 ViT layers  (ViT has 24 layers)
    #   stage 1 → 12 out of 24 LLM layers
    --custom-pipeline-recompute-layers 12,12  # ViT 24layers

    --recompute-granularity full
    --recompute-method uniform
    ...
)

MODEL_PARALLEL_ARGS=(
    --pipeline-model-parallel-size 2   # PP=2
    --tensor-model-parallel-size  1   # TP=1
    ...
)

For PP=3 (stage 0 holds ViT only, stages 1-2 each hold 18 LLM layers):

TRAINING_ARGS=(
    ...
    --custom-pipeline-layers 0,18,18

    # stage 0 → 12 out of 24 ViT layers  (ViT 24layers)
    # stage 1 → 9 out of 18 LLM layers
    # stage 2 → 9 out of 18 LLM layers
    --custom-pipeline-recompute-layers 12,9,9  # ViT 24layers

    --recompute-granularity full
    --recompute-method uniform
    ...
)

Checkpoint Conversion with Custom Pipeline Layers

The conversion scripts accept the layer split as a positional argument. You must pass the same CUSTOM_PIPELINE_LAYERS value that was used (or will be used) during training.

HuggingFace → Megatron-Core

# 2B  TP=1 PP=2  (custom split 12,16)
AIAK_TRAINING_PATH=/workspace/LLaVA-OneVision-2 \
bash examples/llava_onevision2/convert/convert_2b_hf_to_mcore.sh \
    /path/to/hf_checkpoint  /path/to/mcore_tp1_pp2  1 2 12,16

# 2B  TP=1 PP=4  (custom split 0,9,9,10)
AIAK_TRAINING_PATH=/workspace/LLaVA-OneVision-2 \
bash examples/llava_onevision2/convert/convert_2b_hf_to_mcore.sh \
    /path/to/hf_checkpoint  /path/to/mcore_tp1_pp4  1 4 0,9,9,10

# 4B  TP=1 PP=2  (custom split 12,24)
AIAK_TRAINING_PATH=/workspace/LLaVA-OneVision-2 \
bash examples/llava_onevision2/convert/convert_4b_hf_to_mcore.sh \
    /path/to/hf_checkpoint  /path/to/mcore_tp1_pp2  1 2 12,24

# 4B  TP=1 PP=4  (custom split 0,12,12,12)
AIAK_TRAINING_PATH=/workspace/LLaVA-OneVision-2 \
bash examples/llava_onevision2/convert/convert_4b_hf_to_mcore.sh \
    /path/to/hf_checkpoint  /path/to/mcore_tp1_pp4  1 4 0,12,12,12

# 8B  TP=2 PP=4  (custom split 0,12,12,12)
AIAK_TRAINING_PATH=/workspace/LLaVA-OneVision-2 \
bash examples/llava_onevision2/convert/convert_8b_hf_to_mcore.sh \
    /path/to/hf_checkpoint  /path/to/mcore_tp2_pp4  2 4 0,12,12,12

Megatron-Core → HuggingFace

# 2B  TP=1 PP=2  (custom split 12,16)
AIAK_TRAINING_PATH=/workspace/LLaVA-OneVision-2 \
bash examples/llava_onevision2/convert/convert_2b_mcore_to_hf.sh \
    /path/to/mcore_tp1_pp2/iter_XXXXXXX  /path/to/hf_output  1 2 12,16

# 4B  TP=1 PP=2  (custom split 12,24)
AIAK_TRAINING_PATH=/workspace/LLaVA-OneVision-2 \
bash examples/llava_onevision2/convert/convert_4b_mcore_to_hf.sh \
    /path/to/mcore_tp1_pp2/iter_XXXXXXX  /path/to/hf_output  1 2 12,24

# 4B  TP=1 PP=4  (custom split 0,12,12,12)
AIAK_TRAINING_PATH=/workspace/LLaVA-OneVision-2 \
bash examples/llava_onevision2/convert/convert_4b_mcore_to_hf.sh \
    /path/to/mcore_tp1_pp4/iter_XXXXXXX  /path/to/hf_output  1 4 0,12,12,12

# 8B  TP=2 PP=4  (custom split 0,12,12,12)
AIAK_TRAINING_PATH=/workspace/LLaVA-OneVision-2 \
bash examples/llava_onevision2/convert/convert_8b_mcore_to_hf.sh \
    /path/to/mcore_tp2_pp4/iter_XXXXXXX  /path/to/hf_output  2 4 0,12,12,12

[!NOTE] The CUSTOM_PIPELINE_LAYERS passed to the conversion script must exactly match the value used when the checkpoint was originally saved. A mismatch will cause incorrect weight loading.

Fully Reproducing Guide

[!TIP] More detailed reproduction steps for the complete process will be provided after the dataset upload is completed.

Mid-Training

To improve model training efficiency, we implement offline sample packing:

1. Download the Mid-Training-85M Dataset
2. Pack the data into WebDataset format, refer to Examples offlinepacking and Offline Padding-Free Data Packing

Instruct

1. Download the LLaVA-OneVision-1.5-Instruct-Data
2. Convert the data into WebDataset format, refer to Conversion for Mixed Instruction Data

Roadmap

Q4 2025 Key Deliverables:

1. Ultra-efficient MoE Training
2. Full Video Input LLM

Contributors

Thanks so much to all of our amazing contributors!

Citation

If you find LLaVA-OneVision-1.5 useful in your research, please consider to cite the following related papers:

@inproceedings{LLaVA-OneVision-1.5,
  title={LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training},
  author={An, Xiang and Xie, Yin and Yang, Kaicheng and Zhang, Wenkang and Zhao, Xiuwei and Cheng, Zheng and Wang, Yirui and Xu, Songcen and Chen, Changrui and Wu, Chunsheng and Tan, Huajie and Li, Chunyuan and Yang, Jing and Yu, Jie and Wang, Xiyao and Qin, Bin and Wang, Yumeng and Yan, Zizhen and Feng, Ziyong and Liu, Ziwei and Li, Bo and Deng, Jiankang},
  booktitle={arXiv},  
  year={2025}
 }

@inproceedings{xie2025region,
  title={Region-based Cluster Discrimination for Visual Representation Learning},
  author={Xie, Yin and Yang, Kaicheng and An, Xiang and Wu, Kun and Zhao, Yongle and Deng, Weimo and Ran, Zimin and Wang, Yumeng and Feng, Ziyong and Miles, Roy and Elezi, Ismail and Deng, Jiankang},
  booktitle={ICCV},
  year={2025}
}

@article{lillava,
  title={LLaVA-OneVision: Easy Visual Task Transfer},
  author={Li, Bo and Zhang, Yuanhan and Guo, Dong and Zhang, Renrui and Li, Feng and Zhang, Hao and Zhang, Kaichen and Zhang, Peiyuan and Li, Yanwei and Liu, Ziwei and Li, Chunyuan},
  journal={Transactions on Machine Learning Research}
  year={2024}
}

Acknowledgement

We extend our sincere gratitude to AIAK team of the Baige AI computing platform from Baidu AI Cloud for providing the exceptional training framework. The outstanding capabilities of AIAK-Training-LLM and AIAK-Megatron have significantly accelerated our training process with remarkable efficiency. These cutting-edge frameworks have been instrumental in achieving our research goals. To get full AIAK support, you can contact Baidu Cloud.

We acknowledge the support of Synvo AI for contributing to the partial data annotation in this work, and also thank the maintainers and contributors of the following open-source projects, whose work greatly inspired and supported our research:

• LLaVA: Large Language-and-Vision Assistant — LLaVA
• LLaVA-NeXT: Next-generation multi-modal assistant — LLaVA-NeXT
• lmms-eval: A standardized evaluation framework for Large Multimodal Models — lmms-eval
• Megatron-LM: Efficient, scalable training for large language models — Megatron-LM
• Qwen2.5-VL: Strong vision-language foundation model — Qwen2.5-VL
• InternVL: Open-source large-scale vision-language foundation model — InternVL
• Qwen3: Next-generation Qwen LLM — Qwen
• MetaCLIP: Scalable contrastive pretraining — MetaCLIP
• FineVision: Open Data Is All You Need — FineVision