Dataset Description

This dataset is the training set of ColPali it includes 127,460 query-image pairs from both openly available academic datasets (63%) and a synthetic dataset made up of pages from web-crawled PDF documents and augmented with VLM-generated (Claude-3 Sonnet) pseudo-questions (37%).

Our training set is fully English by design, enabling us to study zero-shot generalization to non-English languages.

Data Curation

We explicitly verify no multi-page PDF document is used both ViDoRe and in the train set to prevent evaluation contamination.

Load the dataset

from datasets import load_dataset
ds = load_dataset("vidore/colpali_train_set", split="train")

Dataset Structure

Here is an example of a dataset instance structure:

features:
  - name: image
    dtype: image
  - name: image_filename
    dtype: string
  - name: query
    dtype: string
  - name: answer
    dtype: string
  - name: source
    dtype: string
  - name: options
    dtype: string
  - name: page
    dtype: string
  - name: model
    dtype: string
  - name: prompt
    dtype: string
  - name: answer_type
    dtype: string

License

All academic datasets used are here redistributed subsampled and under their original license. The synthetic datasets we created with public internet data and VLM synthetic queries are released without usage restrictions.

Citation Information

If you use this dataset in your research, please cite the original dataset as follows:

@misc{faysse2024colpaliefficientdocumentretrieval,
      title={ColPali: Efficient Document Retrieval with Vision Language Models}, 
      author={Manuel Faysse and Hugues Sibille and Tony Wu and Bilel Omrani and Gautier Viaud and Céline Hudelot and Pierre Colombo},
      year={2024},
      eprint={2407.01449},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2407.01449}, 
}