CARAVELA

CARAVELA is a multimodal benchmark for evaluating the Portuguese cultural knowledge of large vision-language models (LVLMs). The official benchmark language is exclusively European Portuguese (pt-PT).

Motivation

Modern LVLMs excel at general-purpose vision-language tasks, but their performance drops sharply on localized, culturally specific content that is underrepresented in global training data. Portugal has a rich cultural heritage — historical monuments, gastronomy, prominent figures, and distinct art — that existing cultural benchmarks largely overlook, leading to systematic biases in models trained predominantly on Western contexts.

CARAVELA was conceived to bridge this gap. By testing models against canonical Portuguese cultural representations, it provides a rigorous framework to quantify cultural alignment and visual understanding across Portugal's cultural elements — each question requires jointly processing an image and European Portuguese text, so it cannot be answered from text alone.

What It Evaluates

Every item is anchored to a culturally relevant Portuguese entity and its image, and is organized along three axes:

• 5 cultural categories: Gastronomy, Monuments, Personalities, Art, Locations.
• 3 knowledge dimensions: Temporal (historical timelines and dates), Cultural (symbolic meaning, societal impact, and traditions), and Spatial (geographical context and physical features).
• 3 task formats: Multiple-Choice (MCQ), Visual Question Answering (VQA), and Reasoning.

This design enables fine-grained diagnostic evaluation — identifying where a model fails and in which cultural domain.

Dataset Structure

CARAVELA contains 12,983 image–question pairs derived from 3,354 unique Portuguese cultural entities.

Schema

Licensing

This dataset is dual-licensed:

• Annotations — the questions, options, and answers were synthetically generated using open-weight models and are released under apache-2.0.
• Images — all images are sourced from Wikimedia Commons / Wikipedia and retain their original licenses. Refer to the corresponding Wikimedia Commons entry for each image's specific terms and attribution requirements.

When using this dataset you must comply with both the annotation license and the original image licenses.

Citation

If you use CARAVELA or AMALIA-VL in your work, please cite:

@article{gloria2026amalia,
    title={AMALIA-VL: A Native European Portuguese Open-Source Vision and Language Model},
    author={Gl{\'o}ria-Silva, Diogo and Cardeira, Jo{\~a}o and da Luz, Manuel Letras and Simpl{\'\i}cio, Afonso and Vinagre, Gon{\c{c}}alo and Tavares, Diogo and Ferreira, Rafael and Calvo, In{\^e}s and Vieira, In{\^e}s and Semedo, David and others},
    journal={arXiv preprint},
    year={2026}
}