OV2 Quickstart

Quickstart bundle for LLaVA-OneVision-2 (OV2). Contains everything needed to reproduce SFT training and run inference: packed SFT data, ready-to-use HF inference model, Megatron-Core checkpoint, and a Megatron training environment snapshot.

Total size: ~374 GB across 329 files.

---

Contents

1. packed_mixed_sft_cap_v30s/ — 308 GB

Packed mixed SFT (image + video + caption) dataset, sharded for distributed training via Megatron-Energon.

• Format: WebDataset shards (.tar + .tar.idx)
• Layout: 4 nodes × 72 shards each

  packed_mixed_sft_cap_v30s/
  ├── dataset.yaml              # Energon Metadataset config
  ├── node_a/webdataset/        # 77 GB — mixed_a-000000.tar … mixed_a-000035.tar (+ .idx)
  ├── node_b/webdataset/        # 78 GB — mixed_b-*
  ├── node_c/webdataset/        # 78 GB — mixed_c-*
  └── node_d/webdataset/        # 77 GB — mixed_d-*
  

• Sample counts (from dataset.yaml): ~508k samples per node, ~2.03M total
• Augmentation: disabled (augmentation: false)

Use with Energon:

from megatron.energon import get_train_dataset, WorkerConfig
ds = get_train_dataset("packed_mixed_sft_cap_v30s/dataset.yaml", ...)

2. ov_encoder_p14m22_qwen3_hf/ — 8.9 GB

HuggingFace-format inference checkpoint for LLaVA-OneVision-2 with Qwen3 LLM backbone.

• Architecture: LlavaOnevision2ForConditionalGeneration
• LLM: Qwen3-4B-Instruct-2507 (hidden_size=2560, intermediate_size=9728)
• Vision encoder: patch-14, m22 variant
• Precision: bfloat16
• Custom modeling code (trust_remote_code required):
  • modeling_llava_onevision2.py
  • configuration_llava_onevision2.py
  • processing_llava_onevision2.py
  • codec_video_processing_llava_onevision2.py
  • video_processing_llava_onevision2.py
• Demo script: demo_inference.py

3. ov_encoder_p14m22_qwen3_mcore_tp1pp1/ — 8.9 GB

Equivalent Megatron-Core checkpoint of the same model, parallel layout TP=1, PP=1. Use this for continued training or fine-tuning in Megatron-LM / NeMo.

ov_encoder_p14m22_qwen3_mcore_tp1pp1/
├── latest_checkpointed_iteration.txt
└── release/
    └── mp_rank_00/
        └── model_optim_rng.pt

4. llava_megatron.26.05.tar — 24 GB

Frozen training environment snapshot (released 2025-05-26, hence 26.05) containing the Megatron-LM fork, dependencies, and tooling used to produce the checkpoints in this repo. Provided as a tarball of an artifact directory (blobs/sha256/... content-addressed layout, 139 entries).

Extract:

tar -xf llava_megatron.26.05.tar

Use this to reproduce results bit-for-bit when external pip/git sources drift.

---

Quickstart

# Download just the inference model
hf download lmms-lab-encoder/ov2_quickstart \
    --repo-type dataset \
    --include "ov_encoder_p14m22_qwen3_hf/*" \
    --local-dir ./ov2

# Or pull everything (374 GB)
hf download lmms-lab-encoder/ov2_quickstart \
    --repo-type dataset \
    --local-dir ./ov2

---

File Manifest