SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing

Mingfei Chen*, Zijun Cui*, Xiulong Liu*, Jinlin Xiang, Caleb Zheng, Jingyuan Li, Eli Shlizerman
* Equal contribution

Overview

3D spatial reasoning in dynamic audio-visual environments remains largely unexplored by current Audio-Visual LLMs (AV-LLMs). SAVVY is a training-free reasoning pipeline that enhances AV-LLMs by recovering object trajectories and constructing a unified global 3D map for spatial question answering.

Data Pipeline

We implement a four-stage pipeline to construct SAVVY-Bench. The stages are Data Preprocessing, Annotation, QA Synthesis, and Quality Review. Each stage combines automated tools with human checks to ensure that every Question–Answer (QA) pair is precise.

Data Preprocessing

We preprocess the video data from the Aria Everyday Activities (AEA) Dataset and integrate raw annotations—such as word-level transcriptions, camera-wearer trajectories, and other sensor signal records—into a unified metadata schema. For video preprocessing, the original fisheye recordings are undistorted into rectilinear frames to ensure compatibility with AV-LLMs. In scenarios with two wearer-mounted camera streams, the videos are temporally aligned to form a unified timeline. This alignment supports consistent segmentation of speech into sentences and facilitates accurate speech topic extraction.

Annotation and Ground Truth Generation

Our annotation focuses primarily on objects and events. Static Object Annotation. Static objects are automatically detected using EFM3D based on a predefined list of object categories (e.g., couch, fireplace). We use Vision-LLM to generate a informative description phrase for each detected object. Annotators then inspect the 3D coordinates and descriptions in a point-cloud viewer, correcting any errors in location, category, or description as needed.

Sounding Event Annotation. For each sound event, we annotate the event description or transcription, its start and end times, and the identity and 3D location of the sound source—if the source is tied to a physical object (e.g., running water with a faucet, a thud with a door). Human annotators adjust the event time span and label the source object and its position accordingly. Specifically for speech events, we first cluster raw word-level transcripts into complete sentences. Annotators then label speech events on a sentence-by-sentence basis. A prompted, rule-based agent converts these validated sentences into concise speech topics that describe individual conversational moments.

QA Synthesis

We use template scripts to generate QA pairs for SAVVY-Bench. These scripts integrate the unified metadata with the new annotations and ground truth data using well-defined question schemas, resulting in unambiguous and structured QA pairs.

SAVVY-Bench Data Preprocessing Utilities

This directory (data_utils/) contains data preprocessing tools for SAVVY-Bench, the first benchmark for dynamic 3D spatial reasoning in audio-visual environments, introduced in SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing. It provides downloading and preprocessing scripts for Aria Everyday Activities Dataset, Meta Reality Labs-R videos used in SAVVY-Bench.

Note: This is part of the SAVVY repository. For the main SAVVY algorithm code, see the parent directory.

Setup Environment

Step 1: Create Conda Environment

conda create -n savvy-bench python=3.10 -y
conda activate savvy-bench

# For AEA pre-processing
pip install requests tqdm numpy scipy opencv-python imageio open3d matplotlib tyro pillow natsort

# Install Project Aria Tools (ESSENTIAL for VRS file processing)
pip install 'projectaria-tools[all]'

# Install PyTorch
# CPU version:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu

# Or GPU (optional):
# conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia

Step 2: Initialize Submodules

Please ensure the egolifter submodule is initialized:

# From the SAVVY repo root
git submodule update --init --recursive

Egolifter: The EgoLifter submodule (data_utils/egolifter/) is unmodified from the original source and is licensed under Apache License 2.0.

Download and Process Aria Everyday Activities Videos

Step 1: Access the Dataset

1. Visit Aria Everyday Activities Dataset, Meta Reality Labs-R
2. Follow the instructions to access the dataset
3. Download the Aria Everyday Activities Dataset.json file and place it in the data_utils/ directory

Expected Download Size: 120.51 GB total

Step 2: Data Download and Undistortion

# Navigate to data_utils directory
cd data_utils

conda activate savvy-bench

# Download AEA data
python scripts/download_aea_videos.py

# Process and undistort
python scripts/process_aea_data.py

# Extract audio
python scripts/extract_vrs_audio.py aea/aea_data aea/aea_processed

# Convert to video
python scripts/process_videos.py aea/aea_processed aea/aea_processed

Note: Audio and video processing automatically trim segments based on timestamps in aea/video_timestamps.txt. The special sequences loc2_script3_seq3_rec1, loc2_script3_seq3_rec2 are split into 4 segments each (seq31, seq32, seq33, seq34).

Step 3: Verify Processing

After completion, you should have the following structure within data_utils/:

Raw Data: data_utils/aea/aea_data/ (53 raw VRS recordings with MPS SLAM data)

Processed Data: data_utils/aea/aea_processed/ with structure:

data_utils/aea/aea_processed/
├── loc1_script2_seq1_rec1/
│   ├── audio/
│   │   └── loc1_script2_seq1_rec1.wav  # Timestamp-trimmed audio (mic 5,6 @ 48kHz)
│   ├── video/
│   │   └── loc1_script2_seq1_rec1.mp4  # Timestamp-trimmed video (20 FPS, undistorted)
│   ├── images/                         # Undistorted RGB frames
│   │   ├── frame_000001.jpg
│   │   └── ...
│   └── transforms.json                 # Camera poses & intrinsics (3DGS format)
├── loc1_script2_seq1_rec2/
│   └── ...
└── [56 total sequences including seq31-34 splits]

Citation

@article{chen2025savvy,
    title={SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing},
    author={Mingfei Chen and Zijun Cui and Xiulong Liu and Jinlin Xiang and Caleb Zheng and Jingyuan Li and Eli Shlizerman},
    year={2025},
    eprint={2506.05414},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}
# This paper will appear in NeurIPS 2025