SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing

Mingfei Chen*, Zijun Cui*, Xiulong Liu*, Jinlin Xiang, Caleb Zheng, Jingyuan Li, Eli Shlizerman
University of Washington * Equal contribution

Overview

3D spatial reasoning in dynamic audio-visual environments remains largely unexplored by current Audio-Visual LLMs (AV-LLMs). SAVVY introduces a training-free reasoning pipeline that enhances AV-LLMs by recovering object trajectories and constructing unified global 3D maps for spatial question answering. Alongside the algorithm, we present SAVVY-Bench, the first benchmark for evaluating dynamic 3D spatial reasoning in audio-visual environments.

⭐ If this project helps your research, a star is appreciated!

Project Components

This project consists of three main components, each hosted in separate repositories:

Quick Links

---

SAVVY Algorithm

Main Algorithm Repository - Core SAVVY pipeline implementation

The SAVVY algorithm employs a multi-stage reasoning approach:

• Snapshot Descriptor (SD): Prompts AV-LLMs to extract structured descriptions and initial object trajectories
• Text-Guided Segmentation (Seg): Refines trajectories using visual foundation models (CLIPSeg, SAM2, ZoeDepth)
• Spatial Audio & Track Aggregation: Fuses multi-modal cues into a unified 3D map for final predictions

Get Started: Visit the SAVVY repository for installation, usage instructions, and algorithm details.

---

SAVVY-Bench Dataset

Hugging Face Repository - Benchmark dataset, annotations, and data preprocessing tools

SAVVY-Bench is constructed through a four-stage pipeline combining automated tools with human validation:

1. Data Preprocessing: Undistorts videos, aligns multi-stream recordings, processes sensor data
2. Annotation: Objects and sounding events with 3D spatial annotations
3. QA Synthesis: Template-based generation of structured question-answer pairs
4. Quality Review: Human verification ensuring precision

Access Dataset:

• Download from Hugging Face
• Data preprocessing scripts available in the SAVVY repository under data_utils/ folder

---

AV-LLM Evaluation Code on SAVVY-Bench

Evaluation Repository - We currently support the evaluation of the following AV-LLMs on SAVVY-Bench:

• VideoLLaMA2
• MiniCPM-o2.6
• Video-SALMONN (video-salmonn-13b)
• Gemini 2.5 Flash / Pro
• Ola
• EgoGPT
• Longvale

Run Benchmarks: Visit the SAVVY-Bench evaluation repository for setup instructions and evaluation scripts.

Citation

If you use SAVVY or SAVVY-Bench in your research, please cite:

@inproceedings{NEURIPS2025_ac895e51,
 author = {Chen, Mingfei and Cui, Zijun and Liu, Xiulong and Xiang, Jinlin and Zheng, Yang and Li, Jingyuan and Shlizerman, Eli},
 booktitle = {Advances in Neural Information Processing Systems},
 editor = {D. Belgrave and C. Zhang and H. Lin and R. Pascanu and P. Koniusz and M. Ghassemi and N. Chen},
 pages = {118999--119038},
 publisher = {Curran Associates, Inc.},
 title = {SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing},
 url = {https://proceedings.neurips.cc/paper_files/paper/2025/file/ac895e51849bfc99ae25e054fd4c2eda-Paper-Conference.pdf},
 volume = {38},
 year = {2025}
}

License

Please refer to individual component licenses:

• SAVVY Algorithm: See LICENSE
• Original Aria Dataset: Project Aria License Agreement
• EgoLifter submodule: Apache License 2.0

---