Turning Generators into Retrievers: Unlocking MLLMs for Natural Language-Guided Geo-Localization

Official implementation of "Turning Generators into Retrievers: Unlocking MLLMs for Natural Language-Guided Geo-Localization" (CVPR 2026, Findings).

This repository provides a framework for leveraging Multimodal Large Language Models (MLLMs) for precise, natural language-guided geo-localization by transforming generative capabilities into retrieval strengths. Checkpoint weights are avaiable in the Preparation section.

---

📋 Table of Contents

• Repository Structure
• Setup & Installation
• Preparation
• Usage Guide
• Configuration & Data Fields
• Citation
• Contact

---

📂 Repository Structure

.
├── train.py              # Main training for model training and fine-tuning
├── eva.py                # Evaluation for CVGText and GeoText
├── eva_compute.py        # (GeoText only) Large-scale metric calculator
├── src/                  # Core source: model definitions, losses, data utilities
└── config/               # YAML configs for each model architecture
    ├── internvl.yaml
    └── smolvlm.yaml

eva_compute.py is required for GeoText evaluation. Running metric computation inside eva.py on the full GeoText split will cause Out-Of-Memory (OOM) errors on standard GPUs.

---

🛠 Setup & Installation

Prerequisites

• Python 3.10+
• Conda (recommended)
• CUDA-compatible GPU

1. Create Environment

conda create -n ngcg python=3.10
conda activate ngcg
pip install -r requirements.txt

---

📁 Preparation

Data Structure

Organize your data directory as follows:

data/
├── CVGText/
│   ├── images/
│   │   └── <city>/             
│   ├── texts/
│   │   └── <city>/
│   ├── text2sate_<city>_train.json
│   └── text2sate_<city>_test.json
│   ├── text2osm_<city>_train.json
│   └── text2osm_<city>_test.json
└── GeoText/
    ├── images/
    │   ├── train/
    │   └── test/  
    ├── geo_t2i_train.json
    └── geo_t2i_test.json

Processed Dataset Json Document Download

All JSON files follow the original dataset author's format. We augment each entry with four additional fields to unify the retrieval pipeline — see Configuration & Data Fields for details. We also provide the processed JSON documents via this OneDrive folder with Code:123456

Checkpoint Download

checkpoint for InternVL (including GeoText and CVGText) can be download via this OneDrive link with Code: 123456

---

🚀 Usage Guide

1. Training

Submit a training job on a SLURM-managed cluster:

sbatch train_internvl_geo.sh

2. Evaluation

Evaluation is handled differently depending on dataset scale.

---

A. CVGText (Standard)

eva.py runs inference and computes metrics in a single pass:

python eva.py \
    --config_path        <path/to/config.yaml> \
    --checkpoint_to_eval <path/to/checkpoint> \
    --dataset_name       CVGText \
    --subset_name        vec_text2sate_newyork \
    --augmentation       'none'

---

B. GeoText (Large-Scale, Two-Stage)

Due to the scale of GeoText, evaluation is split into two stages.

Stage 1 — Extract Features

python eva.py \
    --config_path        <path/to/config.yaml> \
    --checkpoint_to_eval <path/to/checkpoint> \
    --dataset_name       GeoText \
    --task               text2image \
    --subset_name        geo_24G_t2i \
    --augmentation       'none'

Stage 2 — Compute Recall@K Metrics

python eva_compute.py \
    --checkpoint_to_eval <path/to/checkpoint> \
    --dataset            GeoText \
    --subset             geo_t2i \
    --task               image2text

---

⚙️ Configuration & Data Fields

We augment JSON annotation files with four additional fields to support the our framework. If the query is text-to-image, you would feed the text into qry_text_field and the corresponding ground-truth image into pos_img_path_field.

These fields are consistent across both CVGText and GeoText and are required for the data loader to function correctly.

---

📜 Citation

If you find this work useful, please cite:

@inproceedings{chen2026turning,
  title     = {Turning Generators into Retrievers: Unlocking MLLMs for Natural Language-Guided Geo-Localization},
  author    = {Yuqi Chen and Xiaohan Zhang and Ahmad Arrabi and Waqas Sultani and Chen Chen and Safwan Wshah},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Findings},
  year      = {2026}
}

---

📧 Contact

For questions about the paper or code, please open a GitHub issue or contact Yuqi Chen by email [email protected].