This repository contains the code for evaluating the AMALIA-VL model, supporting the evaluations reported in:

• AMALIA-VL: A Native European Portuguese Open-Source Vision and Language Model — arXiv:2606.19100
• PorTEXTO: A European Portuguese Benchmark for Visual Text Extraction — arXiv:2606.19096

amalia-vl-eval

This repository includes amalia-vl-eval, a benchmark suite for evaluating Large Vision-Language Models with a focus on European Portuguese (PT-PT).

The suite groups 18 leaf tasks under the tag amalia_vl_eval_pt and spans diagram and chart understanding, document and infographic VQA, scene-text and OCR, perception and college-level multimodal reasoning, visual mathematics, real-world and embodied spatial reasoning, object-hallucination probing, captioning, and referring-expression grounding. The PorTEXTO (visual text extraction) and CARAVELA (cultural visual entities) benchmarks are separate native pt-PT groups with their own tags, described below.

The supported tasks are the following:

PorTEXTO is a separate, native pt-PT group (tag portexto, aggregating group portexto_pt, run with --tasks portexto_pt) and is not part of the amalia_vl_eval_pt suite:

CARAVELA is a separate, native pt-PT group (tag caravela, run with --tasks caravela) and is not part of the amalia_vl_eval_pt suite:

PorTEXTO

PorTEXTO is the first benchmark for contemporary, culturally relevant pt-PT visual text extraction. It is built from frontier-LVLM transcriptions reviewed by native speakers, and spans four subsets — handwritten, handwritten_full_page, in_the_wild, and synthetic — so you can see the characteristic drop in quality from synthetic to real-world text.

Scoring uses ANLS and BLEU-1 (unigram, via sacrebleu). The four subsets also carry the portexto tag, and the aggregating group portexto_pt reports the mean of both metrics.

CARAVELA

CARAVELA (Cultural Awareness and Recognition Assessment for Visual Entity Literacy and Analysis) is a native European Portuguese benchmark probing knowledge of culturally relevant visual entities. It spans three subtasks — caravela_mcq (4-option multiple choice), caravela_vqa (short open-ended answers), and caravela_reasoning (step-by-step reasoning).

Scoring uses accuracy, normalised exact/contains match, and token-level F1 respectively. The three subtasks also carry the caravela tag, and the aggregating group caravela reports the macro-average across them.

Setup

lmms-eval requires Python ≥ 3.10. Using a conda environment (the project default):

conda create -n amalia-vl-eval python=3.12 -y
conda activate amalia-vl-eval

git clone https://github.com/AMALIA-LLM/amalia-vl-eval.git
cd amalia-vl-eval
pip install -e ".[all]"

Running Evaluations

After activating the environment, evaluate the AMALIA-VL checkpoint on the full pt-PT suite. How you launch depends on how many GPUs you have.

Single GPU

Use the standard lmms-eval entry point and pass device_map=auto so the model is placed on the GPU:

python -m lmms_eval \
  --model llava_hf \
  --model_args pretrained=amalia-llm/AMALIA-VL-SFT,device_map=auto \
  --tasks amalia_vl_eval_pt \
  --batch_size 1 \
  --output_path ./results/ \
  --log_samples

You can also pass a single task or a comma-separated list. For example, to run only PorTEXTO on a single GPU:

python -m lmms_eval \
  --model llava_hf \
  --model_args pretrained=amalia-llm/AMALIA-VL-SFT,device_map=auto \
  --tasks portexto_pt \
  --batch_size 1 \
  --output_path ./results/ \
  --log_samples

Or to run the full CARAVELA group (its three leaf tasks):

python -m lmms_eval \
  --model llava_hf \
  --model_args pretrained=amalia-llm/AMALIA-VL-SFT,device_map=auto \
  --tasks caravela \
  --batch_size 1 \
  --output_path ./results/ \
  --log_samples

Multiple GPUs

Launch with accelerate for data parallelism across N GPUs (set --num_processes to the number of GPUs). Do not pass device_map here — each process places the model on its own GPU automatically:

accelerate launch --num_processes=8 --main_process_port 12345 -m lmms_eval \
  --model llava_hf \
  --model_args pretrained=amalia-llm/AMALIA-VL-SFT \
  --tasks amalia_vl_eval_pt \
  --batch_size 1 \
  --output_path ./results/ \
  --log_samples

Note: OPENAI_API_KEY may need to be set to any placeholder value, because the upstream mmmu/mathvision utils eagerly construct an OpenAI judge client at import time. The judge is never called — the pt-PT tasks use rule-based metrics (exact match / accuracy).

Citation

If you use amalia-vl-eval, please cite the AMALIA-VL and PorTEXTO papers:

@misc{amalia-vl-2026,
    title  = {AMALIA-VL: A Native European Portuguese Open-Source Vision and Language Model},
    author = {Diogo Gl{\'o}ria-Silva and Jo{\~a}o Cardeira and Manuel Letras da Luz and
              Afonso Simpl{\'i}cio and Gon{\c{c}}alo Vinagre and Diogo Tavares and
              Rafael Ferreira and In{\^e}s Calvo and In{\^e}s Vieira and David Semedo and
              Jo{\~a}o Magalh{\~a}es},
    year    = {2026},
    eprint  = {2606.19100},
    archivePrefix = {arXiv},
    primaryClass  = {cs.CV},
    url     = {https://arxiv.org/abs/2606.19100}
}

@misc{portexto-2026,
    title  = {PorTEXTO: A European Portuguese Benchmark for Visual Text Extraction},
    author = {Jo{\~a}o Cardeira and Diogo Gl{\'o}ria-Silva and Manuel Letras da Luz and
              Rafael Ferreira and Diogo Tavares and David Semedo and Jo{\~a}o Magalh{\~a}es},
    year    = {2026},
    eprint  = {2606.19096},
    archivePrefix = {arXiv},
    primaryClass  = {cs.CV},
    url     = {https://arxiv.org/abs/2606.19096}
}

Acknowledgements

amalia-vl-eval is a fork of lmms-eval by the EvolvingLMMs-Lab, which is itself a fork of EleutherAI's lm-evaluation-harness. We thank both teams. The pt-PT layer and PorTEXTO benchmark are contributed by the AMALIA project. We recommend you read through the lmms-eval documentation for relevant information on how to add models or tasks.

License

This project inherits the dual license of upstream lmms-eval:

• The main pipeline / structure-related code is under the MIT License, consistent with lm-evaluation-harness.
• The multimodal tasks and models (code under lmms_eval/tasks and lmms_eval/models, including the pt-PT layer and PorTEXTO contributed here) are under the Apache License 2.0.

See LICENSE for the full text of both.