README
PopVQA: Popular Entity Visual Question Answering
PopVQA is a dataset designed to study the performance gap in vision-language models (VLMs) when answering factual questions about entities presented in images versus text.
Paper: https://huggingface.co/papers/2412.14133 Code: https://github.com/ido-co/vlm-modality-gap
🔍 Motivation
PopVQA was curated to explore the disparity in model performance when answering factual questions about an entity described in text versus depicted in an image. This is achieved by asking the same questions twice, once with the textual representation (the entity's name), then, with the visual representation (entity image). We include several questions about every entity to allow a more fine grained evaluation. This dataset was introduced in the paper:
"Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language Models"
Ido Cohen, Daniela Gottesman, Mor Geva, Raja Giryes (2025)
📦 Dataset Structure
The dataset consists of:
entities.csv: Metadata of 15,395 popular entities, of various types (celebrities, landmarks, logos, and paintings).questions.csv: Over 100,000 factual questions, each given in two forms: one referring to a textual representation and one referring to a visual representation of the entity.original_path/: Original images.resized_path/: Images resized to 336×336 with aspect ratio preserved via padding.
entities.csv columns:
| Column | Description |
|---|---|
type | Entity type (e.g., celebs, logos) |
subject | Entity name |
s_uri | Wikidata URI of the subject |
popularity | Wikipedia popularity score |
aliases | Alternate names/aliases for the entity |
image | wiki commons url |
original_path | Path to the original image |
resized_path | Path to the 336x336 padded image |
questions.csv columns:
| Column | Description |
|---|---|
type | Entity type |
subject | Entity name |
question_for_image | Question phrased for visual context (e.g., “...in this image?”) |
question | Textual version of the same question |
possible_answers | List of acceptable answers |
relation | Relation name (e.g., occupation, language) |
s_uri, r_uri, a_uri | Wikidata URIs for subject, relation, and answer |
attribute, a_type | Answer string and attribute types (e.g., "language") |
🏗️ Reproducing the questions.csv File
To regenerate the questions.csv file used in this dataset, you can use the provided script in the scripts/ directory.
📂 Required Structure
Before running the script, make sure your working directory contains:
- A base dataframe CSV with at least the following columns:
s_uri: Wikidata URI of the entitytype: Category of the entity (e.g.,celebs,logos, etc.)
- A folder called
relation_templates/, which should include files named like:relation_templates_celebs.csvrelation_templates_logos.csv- etc.
- Refer to the existing files for reference and structure.
🧪 Generating the Questions
To build the dataset, run:
python scripts/build_dataset.py --base-df path/to/base_entities.csv
Similar repos
Linked in READMEs
README
PopVQA: Popular Entity Visual Question Answering
PopVQA is a dataset designed to study the performance gap in vision-language models (VLMs) when answering factual questions about entities presented in images versus text.
Paper: https://huggingface.co/papers/2412.14133 Code: https://github.com/ido-co/vlm-modality-gap
🔍 Motivation
PopVQA was curated to explore the disparity in model performance when answering factual questions about an entity described in text versus depicted in an image. This is achieved by asking the same questions twice, once with the textual representation (the entity's name), then, with the visual representation (entity image). We include several questions about every entity to allow a more fine grained evaluation. This dataset was introduced in the paper:
"Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language Models"
Ido Cohen, Daniela Gottesman, Mor Geva, Raja Giryes (2025)
📦 Dataset Structure
The dataset consists of:
entities.csv: Metadata of 15,395 popular entities, of various types (celebrities, landmarks, logos, and paintings).questions.csv: Over 100,000 factual questions, each given in two forms: one referring to a textual representation and one referring to a visual representation of the entity.original_path/: Original images.resized_path/: Images resized to 336×336 with aspect ratio preserved via padding.
entities.csv columns:
| Column | Description |
|---|---|
type | Entity type (e.g., celebs, logos) |
subject | Entity name |
s_uri | Wikidata URI of the subject |
popularity | Wikipedia popularity score |
aliases | Alternate names/aliases for the entity |
image | wiki commons url |
original_path | Path to the original image |
resized_path | Path to the 336x336 padded image |
questions.csv columns:
| Column | Description |
|---|---|
type | Entity type |
subject | Entity name |
question_for_image | Question phrased for visual context (e.g., “...in this image?”) |
question | Textual version of the same question |
possible_answers | List of acceptable answers |
relation | Relation name (e.g., occupation, language) |
s_uri, r_uri, a_uri | Wikidata URIs for subject, relation, and answer |
attribute, a_type | Answer string and attribute types (e.g., "language") |
🏗️ Reproducing the questions.csv File
To regenerate the questions.csv file used in this dataset, you can use the provided script in the scripts/ directory.
📂 Required Structure
Before running the script, make sure your working directory contains:
- A base dataframe CSV with at least the following columns:
s_uri: Wikidata URI of the entitytype: Category of the entity (e.g.,celebs,logos, etc.)
- A folder called
relation_templates/, which should include files named like:relation_templates_celebs.csvrelation_templates_logos.csv- etc.
- Refer to the existing files for reference and structure.
🧪 Generating the Questions
To build the dataset, run:
python scripts/build_dataset.py --base-df path/to/base_entities.csv