OpenEarthMap

This is a project for YZV411E course in Istanbul Technical University. It is a distributed inference for satellite land cover classification using tools such as the tool according to Google they call "Google Cloud Dataproc".

Dataset

Here is the OpenEarthMap dataset, according to their authors:

OpenEarthMap is a benchmark dataset for global high-resolution land cover mapping.

Dataset Name and Link according to HuggingFace: EVER-Z/open_earth_map

Quick Start

cd dataproc

# 1. Edit config.yaml with your GCP project and bucket
# (just empty bucket is enough)

# 2. Setup infrastructure (run once)
./scripts/setup_infrastructure.sh

# 3. Create cluster
./scripts/create_cluster.sh

# 4. Submit inference job
./scripts/submit_job.sh 

# Or run parameter sweep across worker counts and machine types
./scripts/run_sweep.sh

# 5. Delete cluster when done
./scripts/delete_cluster.sh

Distributed Inference (Dataproc)

The dataproc/ folder contains scripts for running distributed inference on, as referenced before as well as their authors, Google Cloud Dataproc using the software called "PySpark".

Architecture

According to the softwares and tools defined, as well as their external authors, the architecture and pipeline becomes:

• Spark 4.0 with Arrow for efficient data transfer
• CPU-only inference (cost-effective for batch processing)
• Loads dataset directly from HuggingFace Hub via pyspark-huggingface
• Outputs results to Google Cloud Storage (GCS)

Configuration

All settings are in dataproc/config.yaml:

gcs:
  bucket: "openearthmap-inference"    # GCS bucket for results and model
  project: "your-gcp-project"         # GCP project ID
  model_path: "models/model.pth"      # Path to model weights in GCS
  results_path: "results"             # Output path for inference results
  location: "europe-west1"            # GCS bucket location

cluster:
  name: "openearthmap-inference"      # Dataproc cluster name
  region: "europe-west1"              # Cluster region
  zone: ""                            # Optional: specific zone
  image_version: "3.0.0-RC1-debian12" # Dataproc image version
  
  master:
    machine_type: "c2-standard-8"     # Master node type (8 vCPUs)
    boot_disk_size_gb: 100
    boot_disk_type: "pd-ssd"
  
  workers:
    count: 4                          # Number of worker nodes
    machine_type: "c2-standard-8"     # Worker node type
    boot_disk_size_gb: 100
    boot_disk_type: "pd-ssd"
  
  executor_memory: "23g"              # Spark executor memory

model:
  local_checkpoint: "checkpoints/model.pth"  # Local path for upload

dataset:
  gcs_path: "datasets/oem_val_100parts"      # Dataset location in GCS

secrets:
  huggingface_token: "your-hf-token"  # HuggingFace API token(we will just share it for convenience)

Scripts

Parameter Sweep

Run experiments across different configurations to find optimal settings:

cd dataproc

# Default sweep: 2,4,6 workers × c2-standard-8, c2-standard-4
./scripts/run_sweep.sh

# Custom sweep
./scripts/run_sweep.sh --workers "2 4" --machines "c2-standard-8"

The sweep script:

• Creates clusters with different worker counts and machine types
• Runs inference jobs and collects benchmark results
• Auto-adjusts executor memory based on machine type
• Generates a comparison report with throughput and scaling efficiency
• Resumes from previous runs if interrupted

Local Spark Benchmarking

Run inference locally before scaling to Dataproc:

cd dataproc

# Create local Spark environment
uv venv .venv-spark --python 3.11
source .venv-spark/bin/activate
uv pip install pyspark datasets transformers torch pillow pyyaml psutil numpy

# Benchmark on validation split (500 images)
spark-submit --driver-memory 12g jobs/local_inference_job.py \
    --huggingface \
    --split val \
    --output-path ./results/hf_val_benchmark \
    --num-partitions 4

# Quick test with limited samples
spark-submit --driver-memory 8g jobs/local_inference_job.py \
    --huggingface \
    --split val \
    --limit 50 \
    --output-path ./results/hf_test \
    --num-partitions 2

Environment Variables

Benchmark Results

The following plots show the performance characteristics of distributed inference on "Dataproc":

Throughput Comparison

Scaling Efficiency

Total Job Time

Time Breakdown

Memory Usage

Partition Performance

Citation

According to their respective authors:

@inproceedings{xia2023openearthmap,
  title={OpenEarthMap: A Benchmark Dataset for Global High-Resolution Land Cover Mapping},
  author={Xia, Junshi and others},
  booktitle={WACV},
  year={2023}
}

External Code and Libraries

All external libraries, code and tools used in this project are according to their authors, not us.