VRSBench

VRSBench is a Versatile Vision-Language Benchmark for Remote Sensing Image Understanding. It consists of 29,614 remote sensing images with detailed captions, 52,472 object refers, and 3123,221 visual question-answer pairs. It facilitates the training and evaluation of vision-language models across a broad spectrum of remote sensing image understanding tasks.

Using datasets

from datasets import load_dataset
fw = load_dataset("xiang709/VRSBench", name="VRSBench", split="train", streaming=True)

Attribute explanations in annotation files:

• obj_corner: the coordinates of four object corner points.
• obj_coord: [can be ignored] the bounding box coordinates of objects in the format of [x1,y2,x2,y2], used for annotation generation by GPT-4V inference.
• is_unique: is the object unique within its object category.

Be careful that in our prepare_eval_all.ipynb and provided train/eval json files, box coordinates are normalized to 0-100. If your model does not normalize coordinates, you need to make minor changes accordingly. Note that the original DIOR dataset uses HBB to localize objects, we further convert HBBs to OBBs in our annotations (based on DIOR_R dataset), please check the dior_r branch for annotations.

Dataset curation

To construct our VRSBench dataset, we employed multiple data engineering steps, including attribute extraction, prompting engineering, GPT-4 inference, and human verification.

• Attribute Extraction: we extract image information, including the source and resolution, as well as object information—such as the object category, bounding box, color, position (absolute and relative), and size (absolute and relative)—from existing object detection datasets.
• Prompting Engineering: We carefully design instructions to prompt GPT-4V to create detailed image captions, object referring, and question-answer pairs.
• GPT-4 inference: Given input prompts, we call OpenAI API to automatically generate image captions, object referring, and question-answer pairs based on the prompts.
• Human verification: To improve the quality of the dataset, we engage human annotators to validate each annotation generated by GPT-4V.

Model Training

For the above three tasks, we benchmark state-of-the-art models, including LLaVA-1.5, MiniGPT-v2, Mini-Gemini, and GeoChat, to demonstrate the potential of LVMs for remote sensing image understanding. To ensure a fair comparison, we reload the models that are initially trained on large-scale image-text alignment datasets, and then finetune each method using the training set of our RSVBench dataset. For each comparing method, we finetune the model on the training set of our RSVBench dataset for 5 epochs. Following GeoChat, we use LoRA finetuning to finetune all comparing methods, with a rank of 64.

Social Impact of Dataset

VRSBench provides a comprehensive benchmark for developing and evaluating generalist vision-language models in both remote sensing and computer vision. This dataset not only supports the training and evaluation of advanced vision-language models but also boosts their ability to tackle complex real-world scenarios in remote sensing.

Discussion of Biases

Despite efforts to ensure high-quality annotations through human verification, biases may exist in the interpretations of visual data due to subjective human factors.

Other Known Limitations

Despite VRSBench includes a variety of landscapes, the geographic diversity is limited to the regions covered by the DOTA-v2 and DIOR datasets.

Licensing Information

The dataset is released under the Creative Commons Attribution Non Commercial 4.0, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Future work

Currently, the VRSBench dataset is limited to annotations for RGB images. In future work, we aim to enhance VRSBench by incorporating annotations from a variety of remote sensing data types, including infrared images, multi- and hyperspectral images, Synthetic Aperture Radar (SAR) images, and temporal datasets. This expansion will significantly broaden the dataset’s utility across diverse observation conditions, facilitating more accurate and timely applications in remote sensing.

Citation Information

@misc{li2024vrsbench,
      title={VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding}, 
      author={Xiang Li, Jian Ding, Mohamed Elhoseiny},
      year={2024},
      eprint={xxx},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}  

🙏 Acknowledgement

Our VRSBench dataset is built based on DOTA-v2 and DIOR datasets.

We are thankful to LLaVA-1.5, MiniGPT-v2, Mini-Gemini, and GeoChat for releasing their models and code as open-source contributions.

Contact

xiangli92@ieee.org, KAUST