GEOBench-VLM: Benchmarking Vision-Language Models for Geospatial Tasks

Summary

While numerous recent benchmarks focus on evaluating generic Vision-Language Models (VLMs), they fall short in addressing the unique demands of geospatial applications. Generic VLM benchmarks are not designed to handle the complexities of geospatial data, which is critical for applications such as environmental monitoring, urban planning, and disaster management. Some of the unique challenges in geospatial domain include temporal analysis for changes, counting objects in large quantities, detecting tiny objects, and understanding relationships between entities occurring in Remote Sensing imagery. To address this gap in the geospatial domain, we present GEOBench-VLM, a comprehensive benchmark specifically designed to evaluate VLMs on geospatial tasks, including scene understanding, object counting, localization, fine-grained categorization, and temporal analysis. Our benchmark features over 10,000 manually verified instructions and covers a diverse set of variations in visual conditions, object type, and scale. We evaluate several state-of-the-art VLMs to assess their accuracy within the geospatial context. The results indicate that although existing VLMs demonstrate potential, they face challenges when dealing with geospatial-specific examples, highlighting the room for further improvements. Specifically, the best-performing GPT4o achieves only 40% accuracy on MCQs, which is only double the random guess performance.

Paper GitHub Page Web Page

---

Dataset Structure

Dataset Folders

The dataset consists of multiple folders, each representing a specific subset or task:

• GEOBench-VLM/Single: Contains images and corresponding questions requiring a single image as input.
• GEOBench-VLM/Temporal: Includes images and questions focused on temporal understanding tasks.
• GEOBench-VLM/Ref-Det: Contains task for Referring Expression Detection with relevant images and associated questions.
• GEOBench-VLM/Ref-Seg: Contains Referring Expression Segmentation task with corresponding images and questions.
• GEOBench-VLM/Captioning: Includes images and questions related to Captioning tasks.

Each folder has its own qa.json file that provides questions associated images and answers.

Data Instances

An example of question looks as follows:

{
    "image_path": "images/single_347.bmp",
    "ground_truth": "Nimitz-class aircraft carrier",
    "ground_truth_option": "E",
    "options_list": [
        "Maestrale-class frigate",
        "Murasame-class destroyer",
        "Mega yacht",
        "Kitty Hawk-class aircraft carrier",
        "Nimitz-class aircraft carrier"
    ],
    "options": "A. Maestrale-class frigate   B. Murasame-class destroyer   C. Mega yacht   D. Kitty Hawk-class aircraft carrier   E. Nimitz-class aircraft carrier",
    "prompts": [
        "What type of ship is visible in this image?",
        "Which class or category does the identified vessel belong to?",
        "What is the primary role or purpose of the ship in this image?",
        "Which specific type of military or civilian ship is shown in the scene?",
        "What class of naval or transport vessel is depicted in the given image?"
    ],
    "task": "Ship Type Classification",
    "image_name": "single_347.bmp",
    "question_id": 0,
    "cls_description": ""
}

Data Fields

The data fields are:

- 'image_path': The relative path to the image corresponding to the question.  
- 'file_name': The name of the image file.  
- 'question_id': A unique identifier for each question.  
- 'task': The specific fine-grained task.  
- 'prompts': A list of multiple prompts associated with the question.  
- 'options': Five multiple-choice answers formatted as a single string.  
- 'options_list': A Python list containing the individual options.  
- 'ground_truth': The correct answer for the question.  
- 'ground_truth_option': The index or label of the correct answer.  
- 'cls_description': Additional details about the classes or terminology used in the prompts.  
 

Adding xBD Images to GEOBench-VLM

This repository does not include xBD dataset images by default. You must first download and process the data manually using the following steps:

1 Download the xBD Dataset

Visit https://xview2.org/download and download hold and test sets:

• hold_images_labels_targets.tar.gz
• test_images_labels_targets.tar.gz

2 Extract the Archives

Run the following commands in your terminal to extract the image folders:

mkdir -p xbd_data
tar -xvzf hold_images_labels_targets.tar.gz -C xbd_data
tar -xvzf test_images_labels_targets.tar.gz -C xbd_data

This will result in the following folder structure:

xbd_data/
  ├── hold/
  │    └── images/
  └── test/
       └── images/

3 Add xBD Images to GEOBench-VLM

Now run the script to preprocess the xBD images:

python preprocess_xbd.py xbd_data /path/to/GEOBench-VLM

This will:

• Prefix each image filename with xBD_

• Copy each image to:

  • /path/to/GEOBench-VLM/Single/images/
  • /path/to/GEOBench-VLM/Temporal/images/

---