ReferGPT: Towards Zero-Shot Referring Multi-Object Tracking

Official implementation of "ReferGPT: Towards Zero-Shot Referring Multi-Object Tracking" (CVPRW 2025)

ReferGPT introduces a novel approach to language-guided 3D multi-object tracking by integrating CLIP vision-language understanding, Large Language Models (LLMs), and Vision-Language Models (VLMs) for zero-shot referring multi-object tracking. Our method enables tracking objects based on natural language descriptions without requiring task-specific training.

Methodology

🚀 Key Features

• Zero-Shot Language-Guided Tracking: Track objects using natural language descriptions without task-specific training
• Multimodal Architecture: Combines CLIP, LLM, and VLM for comprehensive understanding
• 3D Multi-Object Tracking: Full 3D tracking with Kalman filtering and advanced association
• Multiple Detector Support: Compatible with PV-RCNN, Point-RCNN, SECOND-IOU, CasA, VirConv, and more
• HOTA Evaluation: Comprehensive evaluation using state-of-the-art HOTA metrics

📋 Table of Contents

• Installation
• Quick Start
• Configuration
• Results
• Citation

🛠️ Installation

For detailed installation instructions, including environment setup, data preparation, and troubleshooting, please see our comprehensive Installation Guide.

Quick Setup:

git clone https://github.com/Tzoulio/ReferGPT.git
cd ReferGPT
conda create -n refergpt python=3.12
conda activate refergpt
pip install -r requirements.txt

Requirements:

• Python 3.12+ (tested with 3.12.9)
• CUDA 11.0+ or CUDA 12.0+
• Linux (Ubuntu 18.04+)

For complete setup instructions and data downloads, visit: 📖 Installation Guide

🚀 Quick Start

Basic Usage

1. Configure paths in config file

# Edit the configuration file
nano config/global/cfg_refergpt.yaml

Update the following paths:

• dataset_path: Path to your KITTI dataset
• detections_path: Path to your detection files
• expression_path: Path to referring expressions
• llm_output_data_file: Path to LLM outputs (if using pre-computed, provided in the Installation guide)

Be aware that the provided LLM outputs are specifically tailored for detections generated by CasA and the parameter settings defined in the configuration file. If new detections are introduced or parameters are changed, the framework will first attempt to match the new detections with existing llm generated captions that we provide (dataset/data/updated_casa_llm_output_data), based on bounding box coordinates (see model/matching/trajectory_matching.py, line 59 and onwards). If no match is found, a new llm caption will be created.

To enable LLM output generation, update the configuration file by setting use_llm: True and save_llm_output: True.

2. Run tracking

# Run tracking for both cars and pedestrians
bash run_tracker.sh

Advanced Usage

Run specific tracking type:

# For cars only
python3 main.py --cfg_file config/global/cfg_refergpt.yaml

# For pedestrians only  
python3 main.py --cfg_file config/global/cfg_refergpt_pedestrian.yaml

Generate new LLM outputs:

# Enable LLM generation in config
# Set use_llm: True and save_llm_output: True in config file
python3 main.py --cfg_file config/global/cfg_refergpt.yaml

Supported VLM Models

• Phi-3-Vision: Lightweight and efficient (default)
• LLaVA: High-quality vision-language understanding
• GPT-4V: Premium performance (requires API key)

📈 Results

Refer-KITTI Benchmark Results

Performance Characteristics

• Zero-shot Capability: No task-specific training required
• Language Flexibility: Supports complex natural language queries
• 3D Accuracy: State-of-the-art 3D tracking performance

🙏 Acknowledgments

This work builds upon the foundation of PC3T. Many thanks for their wonderful works.

📝 Citation

If you find ReferGPT useful in your research, please consider citing:

@misc{chamiti2025refergptzeroshotreferringmultiobject,
      title={ReferGPT: Towards Zero-Shot Referring Multi-Object Tracking}, 
      author={Tzoulio Chamiti and Leandro Di Bella and Adrian Munteanu and Nikos Deligiannis},
      year={2025},
      eprint={2504.09195},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2504.09195}, 
}

📄 License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.