S2-100K

A complete re-host of the S2-100K dataset used to train SatCLIP (Klemmer et al., 2023).

100,000 globally sampled multi-spectral Sentinel-2 L2A image patches (256×256 px, 12 bands, uint16), with longitude/latitude coordinates for each patch.

Why this re-host?

The original Microsoft-hosted blob (https://satclip.blob.core.windows.net/...) is the canonical source. The only public mirror, davanstrien/satclip, is missing some patches (discussion). This repo is a verified-complete copy: 100,000 / 100,000 patches present, sourced directly from the original Microsoft container.

File layout

metadata.parquet        # fn, lon, lat, patch_idx, shard  (100,000 rows)
index.csv               # fn, lon, lat (verbatim from upstream)
data/shard-00000.tar    # patches 0..999
data/shard-00001.tar    # patches 1000..1999
...
data/shard-00099.tar    # patches 99000..99999
satclip.tar             # all 100,000 patches in one tar (78 GB) - included as a safety copy

Each shard is a plain (uncompressed) tar containing 1000 patch_<N>.tif files. Sharding the original 78 GB tar makes resumable downloads and partial loads practical; the single satclip.tar is included as well so users can grab everything at once if they prefer.

Per-patch format

Each patch_<N>.tif is a GeoTIFF with:

• Shape: 256 × 256 pixels
• Bands: 12 (Sentinel-2 L2A: B01, B02, B03, B04, B05, B06, B07, B08, B8A, B09, B11, B12 — B10 is excluded)
• dtype: uint16 (raw reflectance, scale by 10,000 to get [0,1])
• CRS: UTM zone of the source raster (EPSG:326XX / 327XX)
• Resolution: 10 m

Note: Pillow / datasets.Image cannot decode 12-band uint16 GeoTIFFs. Use rasterio (or tifffile) to read these files.

Loading

Stream a few shards directly

import io
import tarfile

import rasterio
from huggingface_hub import hf_hub_download

shard_path = hf_hub_download(
    repo_id="kklmmr/s2-100k",
    filename="data/shard-00000.tar",
    repo_type="dataset",
)

with tarfile.open(shard_path) as tar:
    member = tar.getmember("patch_42.tif")
    buf = tar.extractfile(member).read()

with rasterio.MemoryFile(buf) as mem, mem.open() as src:
    arr = src.read()           # shape: (12, 256, 256), dtype uint16
    bounds = src.bounds        # UTM bbox
    crs = src.crs              # e.g. EPSG:32610

Pair patches with their coordinates

import pandas as pd
from huggingface_hub import hf_hub_download

meta_path = hf_hub_download(
    repo_id="kklmmr/s2-100k",
    filename="metadata.parquet",
    repo_type="dataset",
)
meta = pd.read_parquet(meta_path)
# columns: fn, lon, lat, patch_idx, shard

Download everything

from huggingface_hub import snapshot_download

local_dir = snapshot_download(
    repo_id="kklmmr/s2-100k",
    repo_type="dataset",
    local_dir="./s2-100k",
)

Provenance

• Original dataset: Sampled and released by the SatCLIP authors (Klemmer et al., 2023).
• Original host: https://satclip.blob.core.windows.net/$web/satclip/ (Microsoft Azure Blob Storage).
• Source code: microsoft/satclip.
• Paper: SatCLIP: Global, General-Purpose Location Embeddings with Satellite Imagery.

This repository contains only the dataset. The pretrained SatCLIP model checkpoints are available from the original SatCLIP authors.

License

MIT, matching the upstream SatCLIP project.

Citation

@article{klemmer2023satclip,
  title={SatCLIP: Global, General-Purpose Location Embeddings with Satellite Imagery},
  author={Klemmer, Konstantin and Rolf, Esther and Robinson, Caleb and Mackey, Lester and Ru{\ss}wurm, Marc},
  journal={arXiv preprint arXiv:2311.17179},
  year={2023}
}