🛰️ Landsat30-AU

💿 Data Availability

The image data used in this project is stored in a dedicated repository on the Hugging Face Hub. All images are provided as zip files.

Accessing the Data

You can find and download the dataset from the following URL:

• Hugging Face Dataset: https://huggingface.co/datasets/supermarkioner/Landsat30-AU

Please download the zip files from the "Files and versions" tab in the repository and extract them to your designated data directory.

📊 Benchmark Evaluation

To ensure a smooth and error-free benchmark evaluation, we recommend maintaining four distinct Python environments. This approach helps to avoid potential conflicts between the dependencies of different Vision Language Models (VLMs).

🌳 Environment 1: EarthDial

EarthDial requires a dedicated Python environment. Detailed setup instructions can be found at its official GitHub repository:

• EarthDial: https://github.com/hiyamdebary/EarthDial

🌋 Environment 2: RS-LLaVA

Similarly, RS-LLaVA needs its own isolated environment to function correctly. Please refer to the setup guide available at:

• RS-LLaVA: https://github.com/BigData-KSU/RS-LLaVA

✨ Environment 3: Gemma 3

For Gemma 3, a specific transformers environment is necessary. You can find the model and setup details on its Hugging Face page:

• Gemma 3: https://huggingface.co/google/gemma-3-12b-it

📦 Environment 4: Shared Environment for Other VLMs

The remaining VLMs can share a single Python environment. We recommend setting up this shared environment using the guidance provided for Qwen, as it is compatible with the other models in this group.

The following VLMs can be installed in this shared environment:

• Qwen: Qwen/Qwen2.5-VL-7B-Instruct
• LLaVA: Transformers Documentation
• Llama: meta-llama/Llama-3.2-11B-Vision-Instruct
• MiMo: GitHub Repository
• GLM-V: THUDM/GLM-4.1V-9B-Thinking

📝 Evaluation Input Files

The following CSV/JSON files are required for the evaluation process and contain the ground truth data for various tasks.

Note: The fine-tuning sets (_ft) are derived from the validation subset, which constitutes 15% of the overall dataset.

• Image Captioning (Test Set):
  • captioning_ft_full.csv
• Image Captioning (Fine-Tune Set):
  • caption_benchmark_dataset_val_ft.jsonl
• Visual Question Answering (Test Set):
  • Landsat30-AU-VQA-test.csv
• Visual Question Answering (Fine-Tune Set):
  • vqa_benchmark_dataset_val_ft.jsonl
• Visual Question Answering (One-Shot Set):
  • one_shot.csv

---

🛠️ Dataset Construction

Landsat Imagery

The generation of the Landsat imagery dataset requires the Digital Earth Australia (DEA) Sandbox environment. [1] This is a specialized platform for analyzing Earth observation data.

Requirements:

1. DEA Sandbox Account: You must first register for a free account to access the necessary environment. Please follow the official guidance to set up your account:

   • Registration: DEA Sandbox Homepage

2. Execution Script: The primary script for processing the imagery is designed to run exclusively within the DEA Sandbox.

   • Script: Landsat_imagery.py

3. Input GeoJSON: This script requires the following GeoJSON file, which defines the geographical areas of interest.

   • GeoJSON File: keep_4_landuse_metadata-categories.geojson