EgoSound (CVPR 2026 Highlight)

Official dataset release for
EgoSound: Benchmarking Sound Understanding in Egocentric Videos

For code, evaluation scripts, and usage instructions, please refer to our GitHub repository:
https://github.com/groolegend/EgoSound

Dataset Details

Dataset Description

EgoSound is a benchmark designed to systematically evaluate egocentric sound understanding in Multimodal Large Language Models (MLLMs).

It is the first dataset explicitly developed to study fine-grained audio-visual reasoning from a first-person perspective. EgoSound captures both:

• Environmental sounds arising from human–object interactions
• Human dialogues that support contextual and causal reasoning

The dataset facilitates research on sound perception, spatial localization, causal inference, and cross-modal reasoning in egocentric scenarios.

EgoSound integrates processed video clips and structured annotations derived from:

• Ego4D
• EgoBlind

For videos originating from Ego4D, we perform clip extraction and filtering as described in the paper. In addition, for both Ego4D and EgoBlind, we extract and provide the corresponding audio tracks to support multimodal evaluation.

---

Dataset Structure

The dataset contains:

1. Processed video clips and corresponding audio tracks
2. Question–Answer annotations

---

Video Clip Structure

EgoBlind.zip

Directory layout:

EgoBlind
    ├── videos
    │   ├── *.mp4
    │   ├── ...
    └── audios
        ├── *.wav
        ├── ...

The EgoBlind portion of the dataset consists of 640 processed video clips and their corresponding audio tracks

Ego4d.zip

Directory layout:

Ego4d
    ├── videos
    │   ├── *.mp4
    │   ├── ...
    └── audios
        ├── *.wav
        ├── ...

The Ego4d portion of the dataset consists of 260 processed video clips and their corresponding audio tracks

Annotation Structure

egoblind.json

This file contains structured QA pairs associated with video clips derived from EgoBlind.

ego4d.json

This file contains structured QA pairs associated with video clips derived from Ego4d.

Example annotation format:

{
    "timestamp": "00:00 - 00:05",
    "context": "The video starts with a clear, moderate-volume male voiceover in Mandarin providing instructions: \"When you arrive near the station, if you need help, you can contact the staff for assistance to enter.\"",
    "question_type": "Inferential & Contextual Causality",
    "question": "Why is the male voiceover giving instructions about contacting staff at the start of the video?",
    "answer": "To guide passengers who may need assistance entering the station when they arrive near it.",
    "video_path": "EgoBlind/videos/00923.mp4",
    "question_id": "00923_1"
}

Intended Uses

EgoSound is intended for:

• Benchmarking multimodal large language models
• Studying egocentric audio-visual reasoning
• Evaluating sound perception and causal understanding
• Cross-modal reasoning research

It can be used to evaluate:

• Audio-only models
• Video-only models
• Audio-video multimodal models

---

Citation

If you use EgoSound in your research, please cite:

@misc{zhu2026egosoundbenchmarkingsoundunderstanding,
      title={EgoSound: Benchmarking Sound Understanding in Egocentric Videos}, 
      author={Bingwen Zhu and Yuqian Fu and Qiaole Dong and Guolei Sun and Tianwen Qian and Yuzheng Wu and Danda Pani Paudel and Xiangyang Xue and Yanwei Fu},
      year={2026},
      eprint={2602.14122},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2602.14122}
}

---

Contact

For questions, please open an issue in the GitHub repository:

https://github.com/groolegend/EgoSound