Jina VDR (Jina Visual Document Retrieval)

📚 Blogs: Jina VDR | Resolution Experiments

Scope of this Benchmark

Jina VDR is a multilingual, multi-domain benchmark for visual document retrieval. In contrast to other VDR benchmarks that focus on question answering and OCR-related tasks, it has an expanded scope of visual document benchmarking: more query types, a much more diverse array of materials (e.g. maps, markdown documents, advertisements, scans of historical documents), and includes tasks available in many languages.

The code is based on the vidore-benchmark code.

How to run the Evaluation

To evaluate models on the Jina VDR benchmark, install the requirements corresponding to your selected model:

Installation

# For Jina Embeddings v4
pip install ".[jina-v4]"

# For Jina Embeddings v3
pip install ".[jina-v3]"

# For BM25
pip install ".[bm25]"
# Additionally install stopwords if you haven't already
python -c "import nltk; nltk.download('stopwords')"

# For Jina CLIP
pip install ".[jina-clip]"

# For Colpali
pip install ".[colpali-engine]"

# For DSE-Qwen2b
pip install ".[dse]"

Running Evaluation

Use the base command below with the appropriate model configuration:

vidore-benchmark evaluate-retriever \
    --model-class <MODEL_CLASS> \
    --model-name <MODEL_NAME> \
    --collection-name jinaai/jinavdr-visual-document-retrieval-684831c022c53b21c313b449 \
    --dataset-format qa \
    --split test

Model Configurations

Example for Jina Embeddings v4:

# For single vector
vidore-benchmark evaluate-retriever \
    --model-class jev4 \
    --model-name jinaai/jina-embeddings-v4 \
    --collection-name jinaai/jinavdr-visual-document-retrieval-684831c022c53b21c313b449 \
    --dataset-format qa \
    --max-pixels 37788800 \
    --batch-query 1 \
    --batch-passage 1 \ 
    --batch-score 1 \
    --languages ar,bn,de,en,es,fr,hi,hu,id,it,jp,ko,my,nl,pt,ru,th,ur,vi,zh \
    --split test
    
# For multi vector"
vidore-benchmark evaluate-retriever \
    --model-class jev4 \
    --model-name jinaai/jina-embeddings-v4 \
    --collection-name jinaai/jinavdr-visual-document-retrieval-684831c022c53b21c313b449 \
    --dataset-format qa \
    --max-pixels 37788800 \
    --batch-query 1 \
    --batch-passage 1 \ 
    --batch-score 1 \
    --split test \
    --languages ar,bn,de,en,es,fr,hi,hu,id,it,jp,ko,my,nl,pt,ru,th,ur,vi,zh \
    --vector-type multi_vector

Parameters:

--max-pixels Sets the maximum number of pixels allowed per image during encoding, typically max_vision_tokens * 28 * 28. This parameter helps control input size during image preprocessing and encoding.
The Jina Embeddings v4 results in the paper were generated using --max-pixels 37788800.

Overview of the Dataset Collection

---

Citation

jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval

@misc{günther2025jinaembeddingsv4universalembeddingsmultimodal,
      title={jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval}, 
      author={Michael Günther and Saba Sturua and Mohammad Kalim Akram and Isabelle Mohr and Andrei Ungureanu and Sedigheh Eslami and Scott Martens and Bo Wang and Nan Wang and Han Xiao},
      year={2025},
      eprint={2506.18902},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2506.18902}, 
}