TEOChat: Large Language and Vision Assistant
for Temporal Earth Observation Data [ICLR 2025]

If you like our project, please give us a star ⭐ on GitHub and watch 👀 for the latest updates.

📰 News

• [2025.3.25] 📊 Validation code for the detection datasets and code for fine-tuning TEOChat is now available! Please see TRAIN_AND_VALIDATE.md for validation and fine-tuning instructions.
• [2025.3.10] 🔍 Inference and validation code (for classification datasets) is now available! Please see the Inference section for inference code and TRAIN_AND_VALIDATE.md for validation instructions.
• [2025.1.22] 🔥 Our work has been accepted to ICLR 2025!
• [2024.11.11] 🎉 The TEOChatlas Dataset is now available, along with model training code with instructions described in TRAIN_AND_VALIDATE.md! Validation code and instructions will be provided soon.
• [2024.10.9] 👀 Paper, TEOChat Model, and 🤗 Hugging Face demo are available! Please feel free to watch 👀 this repository for the latest updates.

Table of Contents

• Highlights
• Demo
• Main Results
• Requirements and Installation
• Training & Validating
• Inference
• License
• Acknowledgement
• Citation

😮 Highlights

TEOChat is the first language and vision assistant that can engage in conversation about sequences of temporal earth observation imagery, and exhibits impressive performance on multiple temporal instruction-following tasks.

📚 TEOChatlas: A new instruction-following dataset for temporal EO data

We introduce a new instruction-following dataset for temporal EO data called TEOChatlas which we use to train TEOChat. TEOChatlas contains 554,071 examples spanning dozens of temporal instruction-following tasks.

🤖 TEOChat: A new vision-language model for temporal EO data

We design TEOChat to use a LLaVA-style architecture, combining a temporally shared vision encoder with a LLaMA 2 LLM connected through an MLP vision-language projector

🤗 Demo

Gradio Web UI

We provide an online demo in Huggingface Spaces.

You can also run the demo locally by running the following command:

python videollava/serve/teochat_demo.py 

🚀 Main Results

We demonstrate that TEOChat:

• outperforms a previous VLM for single EO images (GeoChat) and a VLM for temporal sequences of natural images (Video-LLaVA), and also rivals specialist models on multiple tasks.
• achieves impressive zero-shot performance on an EO change detection and a change QA dataset.
• outperforms two strong proprietary foundation models for modeling sequences of images (GPT-4o and Gemini-1.5 Pro).
• possesses strong single image capabilities, outperforming GeoChat on multiple zero-shot scene classification and visual question answering tasks.

Temporal Tasks

Zero-shot Temporal Tasks and Comparison with Proprietary Foundation Models

Single Image Tasks

🛠️ Requirements and Installation

• Python >= 3.9
• Pytorch == 2.2.1
• CUDA Version >= 12.1
• Install required packages:

git clone https://github.com/ermongroup/TEOChat.git
cd TEOChat
conda create -n teochat python=3.9 -y
conda activate teochat
pip install --upgrade pip  # enable PEP 660 support
pip install -e .
pip install git+https://github.com/facebookresearch/pytorchvideo

🗝️ Training & Validating

The training & validating instructions, including how to download the TEOChatlas dataset, are in TRAIN_AND_VALIDATE.md.

🧠 Inference

You can use the following code to run inference with TEOChat on GPU:

from videollava.eval.eval import load_model
from videollava.eval.inference import run_inference_single

tokenizer, model, processor = load_model(model_path="jirvin16/TEOChat", model_base=None, load_8bit=True, device='cuda')

# A list of image paths, all are fed into the model.
image_paths = ["videollava/serve/examples/xBD_cls_1.png", "videollava/serve/examples/xBD_cls_2.png"]
# Note you must include the video tag <video> in the input string otherwise the model will not process the images.
inp = "These are two satellite images in chronological order: <video> Classify the level of damage experienced by the building at location [0, 8, 49, 53]."

response = run_inference_single(model, processor, tokenizer, inp, image_paths)
print(response)

👍 Acknowledgement

• Video-LLaVA The codebase and model we built upon.
• GeoChat The single image instruction-following dataset we included in TEOChatlas.

🔒 License

• The majority of this project is released under the Apache 2.0 license as found in the LICENSE file.
• The service is a research preview intended for non-commercial use only, subject to the model License of LLaMA, Terms of Use of the data generated by OpenAI, and Privacy Practices of ShareGPT. Please contact us if you find any potential violation.

✏️ Citation

If you find our paper and code useful in your research, please consider giving a star :star: and citation :pencil:.

@inproceedings{irvin2024teochat,
  title={TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Data},
  author={Irvin, Jeremy Andrew and Liu, Emily Ruoyu and Chen, Joyce Chuyi and Dormoy, Ines and Kim, Jinyoung and Khanna, Samar and Zheng, Zhuo and Ermon, Stefano},
  booktitle={International Conference on Learning Representations},
  year={2025}
}