MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models

🌐 Website | GitHub | 🏆 Leaderboard | 📖 MMEB-V3 Paper | 📖 MMEB-V2 Paper | 📖 MMEB-V1 Paper | 🤗 Models

Introduction

MMEB-V3 is a comprehensive benchmark for evaluating omni-modality embedding models across text, image, video, audio, visual-document, and agent-centric retrieval scenarios.

Building upon MMEB-V1 and MMEB-V2, MMEB-V3 adds 111 new tasks, resulting in 190 evaluation tasks in total.

MMEB-V3 introduces three major evaluation categories:

• Audio Tasks: audio classification, cross-modal audio retrieval, and audio temporal grounding.
• Text Retrieval Tasks: instruction-following retrieval, reasoning retrieval, long-context retrieval, multi-condition retrieval, and general text retrieval.
• Agent Tasks: tool retrieval, GUI control, and agent memory retrieval.

MMEB-V3 also introduces OmniSET (Omni-modality Semantic Equivalence Tuples), a diagnostic evaluation set containing semantically equivalent instances across text, image, video, and audio. OmniSET enables controlled analysis of modality effects and instruction-conditioned cross-modal retrieval behavior.

The benchmark is designed to measure whether embedding models can reliably follow modality-specific instructions, such as retrieving an image, video, audio clip, visual document, tool, GUI element, or memory item under explicit task constraints.

This Hugging Face repository contains the compressed assets and metadata required to prepare the MMEB-V3 evaluation data.

🚀 What's New

• [2026.04] Released the MMEB-V3 paper.
• Added 111 new tasks, expanding MMEB to 190 tasks in total.
• Added evaluation support for audio, text retrieval, and agent-centric tasks.
• Added OmniSET for controlled omni-modality analysis.
• Unified MMEB-V1, MMEB-V2, and MMEB-V3 assets under a single evaluation data root.
• Added automated dataset preparation and integrity checking through dataset_setup_v3.py.

Benchmark Scope

Guide to MMEB-V3 Data

Please review this section before running evaluation.

• Dataset assets are downloaded from this Hugging Face repository.
• Evaluation metadata and task configurations are provided through the VLM2Vec repository.
• Compressed raw assets are stored under directories using underscores, such as video_tasks and audio_tasks.
• Evaluation-ready directories use hyphens, such as video-tasks and audio-tasks.
• The preparation script extracts and reorganizes downloaded assets into the directory structure expected by the evaluation code.
• MMEB-V1, MMEB-V2, and MMEB-V3 evaluation assets should be placed under one common MMEB-V3 root directory.
• For video evaluation, the provided sampled frames are sufficient for the standard evaluation pipeline.

Dataset Overview

Download

Install the Hugging Face CLI if needed:

pip install -U "huggingface_hub[cli]"

Download MMEB-V3:

export MMEB_V3_ROOT=/path/to/MMEB-V3

hf download VLM2Vec/MMEB-V3 \
  --repo-type dataset \
  --local-dir "$MMEB_V3_ROOT"

The downloaded dataset contains compressed assets and lightweight metadata.

Prepare the Evaluation Data

Clone the VLM2Vec repository:

git clone https://github.com/TIGER-AI-Lab/VLM2Vec.git
cd VLM2Vec

Run the MMEB-V3 setup script:

python experiments/public/data/dataset_setup_v3.py \
  --root "$MMEB_V3_ROOT"

Verify that the expected files and directories were prepared correctly:

python experiments/public/data/dataset_setup_v3.py \
  --root "$MMEB_V3_ROOT" \
  --check-only

The setup script is idempotent:

• It skips completed directories marked with .done.
• It safely extracts TAR and ZIP archives.
• It materializes the directory layout consumed by the evaluation code.
• It checks whether the expected output files and directories exist.

If image-query is not included in the local dataset download, provide an existing copy explicitly:

python experiments/public/data/dataset_setup_v3.py \
  --root "$MMEB_V3_ROOT" \
  --image-query-source /path/to/image-query

No additional argument is required when image-query is already available under the MMEB-V3 root.

Raw Archive Structure

Before running the setup script, the downloaded repository is organized approximately as follows:

MMEB-V3/
├── image_tasks/
│   ├── mmeb_v1.tar.gz
│   └── MCMR.tar.gz
├── audio_tasks/
│   └── *.tar
├── video_tasks/
│   ├── data/
│   └── frames/
│       ├── video_cls.tar.gz
│       ├── video_ret.tar.gz
│       ├── video_mret.tar.gz-*
│       └── video_qa.tar.gz-*
├── visdoc_tasks/
│   ├── visdoc-tasks.data.tar.gz
│   └── visdoc-tasks.images.tar.gz
├── text_tasks/
├── tool_tasks/
├── memory_tasks/
├── gui_tasks/
└── omniset.tar.gz

Directories ending in _tasks contain the downloaded compressed assets. The setup script converts them into the corresponding evaluation-ready -tasks directories.

Evaluation-Ready Structure

After running dataset_setup_v3.py, the expected local directory structure is:

MMEB-V3/
├── image-tasks/
│   ├── MMEB/
│   └── MCMR/
├── image-query/
├── audio-tasks/
├── video-tasks/
│   ├── data/
│   └── frames/
│       ├── video_cls/
│       ├── video_ret/
│       ├── video_mret/
│       └── video_qa/
├── visdoc-tasks/
│   ├── data/
│   └── images/
├── text-tasks/
├── tool-tasks/
├── memory-tasks/
├── gui-tasks/
└── omniset/
    ├── omniset.jsonl
    ├── catalog.jsonl
    ├── val2014/
    ├── videos/
    ├── audios/
    └── frames_omni/

The same root directory can then be passed to the evaluation scripts using --data_basedir.

Evaluation

Evaluation code and task configuration files are available in the VLM2Vec repository.

Standard MMEB-V3 Tasks

The following example evaluates an image-task configuration:

CUDA_VISIBLE_DEVICES=0 python eval.py \
  --pooling mean \
  --normalize true \
  --per_device_eval_batch_size 8 \
  --dataloader_num_workers 1 \
  --model_backbone nvomniembed \
  --model_name /path/to/model \
  --dataset_config experiments/public/eval/image.yaml \
  --encode_output_path exps/vlm2vec/model/image \
  --data_basedir "$MMEB_V3_ROOT"

Replace the model backbone, model path, dataset configuration, output path, and batch size according to your evaluation setup.

The important data argument is:

--data_basedir "$MMEB_V3_ROOT"

OmniSET

Run OmniSET evaluation with:

CUDA_VISIBLE_DEVICES=0 \
MODEL_PATH=/path/to/model \
MODEL_BACKBONE=nvomniembed \
DATA_BASEDIR="$MMEB_V3_ROOT/omniset" \
OUTPUT_PATH=exps/vlm2vec/model/omniset \
PER_DEVICE_EVAL_BATCH_SIZE=8 \
bash experiments/public/eval/eval_omniset.sh

Leaderboard

Model results on MMEB can be found on the official leaderboard:

🏆 MMEB Leaderboard

Please follow the submission instructions on the leaderboard page when submitting a new model result.

Related Resources

• MMEB-V3 Dataset
• MMEB-V2 Dataset
• MMEB-V1 Dataset
• VLM2Vec GitHub Repository
• VLM2Vec Project Website
• VLM2Vec Models
• MMEB Leaderboard

Citation

If you find MMEB-V3 useful in your research, please cite:

@article{huang2026mmeb,
  title={MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models},
  author={Huang, Haohang and Lu, Xuan and Su, Mingyi and Zhang, Xuan and Jiang, Ziyan and Nie, Ping and Zou, Kai and Pfister, Tomas and Chen, Wenhu and Zhang, Wei and others},
  journal={arXiv preprint arXiv:2604.23321},
  year={2026}
}

License

The MMEB-V3 dataset repository is released under the MIT License.

Individual source datasets included or referenced by MMEB-V3 may be subject to their own licenses and terms of use. Users are responsible for complying with the licenses of the corresponding source datasets.