OmniEarth: A Benchmark for Evaluating Vision-Language Models in Geospatial Tasks

Abstract: Vision-Language Models (VLMs) have demonstrated effective perception and reasoning capabilities on general-domain tasks, leading to growing interest in their application to Earth observation. However, a systematic benchmark for comprehensively evaluating remote sensing vision-language models (RSVLMs) remains lacking. To address this gap, we introduce OmniEarth, a benchmark for evaluating RSVLMs under realistic Earth observation scenarios. OmniEarth organizes tasks along three capability dimensions: perception, reasoning, and robustness. It defines 28 fine-grained tasks covering multi-source sensing data and diverse geospatial contexts. The benchmark supports two task formulations: multiple-choice VQA and open-ended VQA. The latter includes pure text outputs for captioning tasks, bounding box outputs for visual grounding tasks, and mask outputs for segmentation tasks. To reduce linguistic bias and examine whether model predictions rely on visual evidence, OmniEarth adopts a blind test protocol and a quintuple semantic consistency requirement. OmniEarth includes 9,275 carefully quality-controlled images, including proprietary satellite imagery from Jilin-1 (JL-1), along with 44,210 manually verified instructions. We conduct a systematic evaluation of contrastive learning-based models, general closed-source and open-source VLMs, as well as RSVLMs. Results show that existing VLMs still struggle with geospatially complex tasks, revealing clear gaps that need to be addressed for remote sensing applications. OmniEarth is publicly available at https://huggingface.co/datasets/sjeeudd/OmniEarth.

---

📂 Data Structure

OmniEarth is organized according to the three-tier hierarchical dimension taxonomy, structured as follows:

├── Perception
│   ├── A1.1-Scene Classification
│   │   ├── images
│   │   └── scene classification.json
│   ├── A1.2-Land-cover Classification
│   │   ├── images
│   │   └── land-cover classification.json
│   ├── A1.3-Image Modality Recognition
│   │   ├── image modality recognition.json
│   │   └── images
│   ├── A1.4-Image Caption
│   │   ├── image caption.json
│   │   └── images
│   ├── A2.1-Visual Grounding
│   │   ├── images
│   │   └── visual grounding.json
│   ├── A2.2-Referring Expression Comprehension
│   │   ├── images
│   │   └── referring expression comprehension.json
│   ├── A2.3-Object Counting
│   │   ├── images
│   │   └── object counting.json
│   ├── A2.4-Fine-grained Category Classification_new
│   │   ├── fine-grained category classification.json
│   │   └── images
│   ├── A2.5-Attribute Recognition
│   │   ├── attribute recognition.json
│   │   └── images
│   ├── A3.1-Referring Expression Segmentation
│   │   ├── images
│   │   ├── masks
│   │   └── referring expression segmentation.json
│   ├── A3.2-Generalized Referring Expression Segmentation
│   │   ├── generalized referring expression segmentation.json
│   │   ├── images
│   │   └── masks
│   └── A3.3-Change Mask Segmentation
│       ├── change mask segmentation.json
│       ├── images
│       └── masks
├── Reasoning
│   ├── B1.1-Spatial Relationship Reasoning
│   │   ├── images
│   │   └── spatial relationship reasoning.json
│   ├── B1.2-Geometric Measurement
│   │   ├── geometric measurement.json
│   │   └── images
│   ├── B1.3-Functional Region Localization
│   │   ├── functional region localization.json
│   │   └── images
│   ├── B2.1-Change Description
│   │   ├── change description.json
│   │   └── images
│   ├── B2.2-Damage Assessment Reasoning
│   │   ├── damage assessment reasoning.json
│   │   └── images
│   ├── B2.3-Long-term Trend Reasoning
│   │   ├── images
│   │   └── long-term trend reasoning.json
│   ├── B2.4-Seasonal Temporal Reasoning
│   │   ├── images
│   │   └── seasonal temporal reasoning.json
│   ├── B3.1-Geo-localization Reasoning
│   │   ├── geo-localization reasoning.json
│   │   └── images
│   ├── B3.2-Disaster Cause Inference
│   │   ├── disaster cause inference.json
│   │   └── images
│   ├── B3.3-Geo-Entity Understanding
│   │   ├── geo-entity understanding.json
│   │   └── images
│   ├── B3.4-Planning Suggestions
│   │   ├── images
│   │   └── planning suggestions.json
│   └── B3.5 City Recognition
│       ├── city recognition.json
│       └── images
└── Robustness
    ├── C1.1-Image Condition Assessment
    │   ├── image condition assessment.json
    │   └── images
    ├── C1.2-Degraded-condition VQA
    │   ├── degraded-condition vqa.json
    │   └── images
    ├── C2.1-Hallucination Detection
    │   ├── hallucination detection.json
    │   └── images
    └── C2.2-Semantic consistency
        ├── images
        └── semantic consistency.json

---

🌰 Example

An example of the Multiple-Choice Question (MCQ) is as follows:

    {
        "image_path": "Perception_A1.1_001.jpg",
        "ground_truth": "golf_course",
        "ground_truth_option": "D",
        "options": "A. race track   B. baseball field   C. amusement park   D. golf course   E. airport   F. stadium",
        "prompts": [
            "What type of scene is depicted in this satellite image?",
            "Identify the category of this landscape.",
            "What is the primary land use shown in this aerial view?",
            "Which kind of recreational or functional area appears in this image?",
            "What is the primary purpose of the area depicted in this image?"
        ],
        "question_id": 1
    }

📦 Download

You can download the dataset using either the Python API or the Command Line Interface (CLI). Choose the method that best fits your workflow.

Method 1: Via Python Code

The easiest way to load the dataset into your project is by using the official Hugging Face datasets library.

First, install the library if you haven't already:

pip install datasets

Then, load the dataset in your Python script:

from datasets import load_dataset

# Load the dataset directly from Hugging Face
dataset = load_dataset("sjeeudd/OmniEarth")

# Optional: Save the dataset to a local directory for offline use
# dataset.save_to_disk("./OmniEarth")

Method 2: Via Command Line (CLI)

If you prefer to download the raw files directly to your local machine, you can use the Hugging Face CLI.

# Install the CLI tool
pip install -U "huggingface_hub[cli]"

# Download the dataset to a specific local directory
huggingface-cli download --repo-type dataset sjeeudd/OmniEarth --local-dir ./OmniEarth

Data Sources and Processing Methods for OmniEarth Benchmark Tasks

The table below provides a detailed overview of the datasets used for each task in the OmniEarth benchmark. It also outlines the data processing methods employed, including label-driven and manual validation, as well as AI-assisted processing for different task categories.

I. Perception Tasks (12 items)

II. Reasoning Tasks (12 items)

III. Robustness Tasks (4 items)