🔥News

• 2025.10.31: Full data for captioning and visual grounding tasks are available on huggingface.
• 2025.07.26: Intern-S1, the most advanced scientific open-source multimodal reasoning model to date, benchmarked on XLRS-Bench.
• 2025.05.28: Training data and finetuned model, code are available at GeoLLaVA-8K.
• 2025.05.16: XLRS-Bench-lite is released on Hugging Face.
• 2025.05.06: XLRS-Bench has been selected for a public competition by China’s Ministry of Education and will be fully released on August 4th, as requested by the organizers.
• 2025.04.04: Selected as Highlight by CVPR 2025!
• 2025.04.01: The dataset is currently under final review and will be released in one month.
• 2025.02.27: XLRS-Bench has been accepted by CVPR 2025!

📚Contents

• 🔥News
• 📚Contents
• 🔍Dataset Overview
• 📸Dataset Examples
• 📜Dataset License
• 🚀Evaluation Pipeline
  • Prompt
  • Huggingface Datasets
  • Evaluation
  • Leaderboard
• 📊Experiment Results
• 📖Citation
• 🙏Acknowledgement
• 📬Contact

🔍Dataset Overview

Remote sensing (RS) images have become essential for monitoring and understanding human environments, driving advancements in applications like precision agriculture, urban planning, and disaster assessment. While recent studies have proposed benchmarks and metrics to assess MLLM performance in RS, these efforts remain limited in image size, annotation method and evaluation dimensions. ​We present XLRS-Bench, a comprehensive benchmark for evaluating the perception and reasoning capabilities of MLLMs in ultra-high-resolution RS scenarios, featuring the largest average image size of 8,500 × 8,500 observed thus far. Our dataset encompasses 45,942 annotations across 16 tasks, all expertly curated by a team of 45 experts. The main advantages of XLRS-Bench compared to existing MLLM benchmarks as follows:

1. Ultra-high Resolution. XLRS-Bench features the largest image sizes available, 10∼20× than that of existing datasets, with 840 images out of all images at a resolution of 10,000×10,000 pixels
2. High-quality Annotation. All the annotations are human involved and manually verified through iterations, resulting in a high-quality benchmark for evaluating MLLMs on real ultra-high-resolution RS scenarios.
3. Comprehensive Evaluation Dimensions: XLRS-Bench covers 10 perception indicators and 6 reasoning dimensions to assess MLLMs’ capabilities, encompassing 16 sub-tasks with a total of 45,942 questions. Especially, XLRS-Bench includes complex reasoning tasks to explore MLLMs’ potential in conducting planning and change detection in long spatial-temporal RS scenarios.

📸Dataset Examples

📜Dataset License

Annotations of this dataset is released under a Creative Commons Attribution-NonCommercial 4.0 International License. For images from:

• DOTA
  RGB images from Google Earth and CycloMedia (for academic use only; commercial use is prohibited, and Google Earth terms of use apply).

• ITCVD
  Licensed under CC-BY-NC-SA-4.0.

• MiniFrance, HRSCD
  Released under IGN’s "licence ouverte".

• Toronto, Potsdam:
  The Toronto test data images are derived from the Downtown Toronto dataset provided by Optech Inc., First Base Solutions Inc., GeoICT Lab at York University, and ISPRS WG III/4, and are subject to the following conditions:

  1. The data must not be used for other than research purposes. Any other use is prohibited.
  2. The data must not be used outside the context of this test project, in particular while the project is still on-going (i.e. until September 2012). Whether the data will be available for other research purposes after the end of this project is still under discussion.
  3. The data must not be distributed to third parties. Any person interested in the data may obtain them via ISPRS WG III/4.
  4. The data users should include the following acknowledgement in any publication resulting from the datasets: “The authors would like to acknowledge the provision of the Downtown Toronto data set by Optech Inc., First Base Solutions Inc., GeoICT Lab at York University, and ISPRS WG III/4.”

Disclaimer:
If any party believes their rights are infringed, please contact us immediately at [email protected]. We will promptly remove any infringing content.

🚀Evaluation Pipeline

Prompt

The common prompt used in VQA follows this format:

[Image] [Question] The choices are listed below:
(A) [Choice A]
(B) [Choice B]
(C) [Choice C]
(D) [Choice D]
Select the best answer for the multiple-choice question based on the image. Only respond with the letter corresponding to the correct answer (A, B, C, D).
The answer is:

or for VQA task Overall Land Use Classification please use:

Select the best answer(s) for the multiple-choice question based on the image. There may be more than one correct option. Only respond with the letter(s) corresponding to the correct answer(s) (A, B, C, D), with multiple choices separated by spaces.

For Visual Grounding and Detailed Image Captioning, please refer to the files in the evaluation folder.

Huggingface Datasets

All available datasets are added to https://huggingface.co/collections/initiacms/xlrs-bench

Evaluation

Using lmms-eval for evaluation:

accelerate launch --num_processes 8 --main_process_port 12345 -m lmms_eval \
    --model "qwen2_5_vl" \
    --model_args "pretrained=Qwen/Qwen2.5-VL-7B-Instruct,use_flash_attention_2=True"  \
    --tasks xlrs-lite \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix qwen2_5_vl_xlrs_lite \
    --output_path ./logs/

Leaderboard

To add your model to our leaderboard, please send your model responses to [email protected]. Refer to the evaluation/samples folder for the required format.

📊Experiment Results

Models are ranked based on their average performance. Proprietary models are highlighted in gray. Task domains are indicated by abbreviations (e.g., “OC” for Overall Counting, “RC” for Regional Counting, etc.).

XLRS-Bench-lite (Avg. stands for macro average across all L3 tasks):

XLRS-Bench (Avg. stands for micro average across all L3 tasks, we will add both micro and macro results soon):

• L-2 performance on VQA tasks.

  

• L-3 perception dimension on VQA tasks.

• L-3 reasoning dimension on VQA tasks.

  

• Visual grounding performance.

  

• Detailed image captioning performance.

  

📖Citation

If you find our work helpful, please consider citing:

@inproceedings{wang2025xlrs,
  title={Xlrs-bench: Could your multimodal llms understand extremely large ultra-high-resolution remote sensing imagery?},
  author={Wang, Fengxiang and Wang, Hongzhen and Guo, Zonghao and Wang, Di and Wang, Yulin and Chen, Mingshuo and Ma, Qiang and Lan, Long and Yang, Wenjing and Zhang, Jing and others},
  booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference},
  pages={14325--14336},
  year={2025}
}

@article{wang2025geollava,
  title={GeoLLaVA-8K: Scaling Remote-Sensing Multimodal Large Language Models to 8K Resolution},
  author={Wang, Fengxiang and Chen, Mingshuo and Li, Yueying and Wang, Di and Wang, Haotian and Guo, Zonghao and Wang, Zefan and Shan, Boqi and Lan, Long and Wang, Yulin and others},
  journal={arXiv preprint arXiv:2505.21375},
  year={2025}
}

🙏Acknowledgement

• [MME-RealWorld] MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?

📬Contact

For any other questions please contact:

• Fengxiang Wang at [email protected]
• Mingshuo Chen at [email protected]