IMAVB is the benchmark of the NeurIPS 2026 paper Senses Wide Shut: A Representation–Action Gap in Omnimodal LLMs. It tests whether an omnimodal model notices when a question makes a false claim about what the model sees or hears.
📄 Paper · 🌐 Project page · 💻 Code
videos/.| Standard premise (answer A–D) | Misleading premise (answer E or F) | |
|---|---|---|
| Vision | standard_vision (500) | misleading_vision (500) |
| Audio | standard_audio (500) | misleading_audio (500) |
A misleading question copies its standard twin and changes exactly one premise detail, such as a shirt colour or the kind of music that plays. Every question lists the same six options, where E is "The visual detail in the question is incorrect" and F is "The audio detail in the question is incorrect", so the right answer to a misleading question is to reject the premise. Each item also carries a question category, a 10-second answer timestamp and, for misleading items, the kind of detail that was changed. We checked all 2,000 items by hand while watching each clip with sound.
from datasets import load_dataset
ds = load_dataset("ngqtrung/IMAVB", split="misleading_audio")
row = ds[0]
print(row["question"], row["correct_answer"], row["video"]) # video path inside this repo
The annotations are released under CC BY-NC-SA 4.0. The clips come from public movie-scene channels (@BingeSociety, @BoxofficeMoviesScenes) and Condensed Movies. They remain the property of their copyright holders and are shared for non-commercial research and evaluation only. If you own a clip and want it removed, please open a discussion on this repository.
@inproceedings{nguyen2026senses,
title = {Senses Wide Shut: A Representation--Action Gap in Omnimodal {LLM}s},
author = {Nguyen Quang, Trung and Gao, Yiming and Pu, Fanyi and Zhang, Kaichen and Sun, Shuo and Liu, Ziwei},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
year = {2026}
}
IMAVB is the benchmark of the NeurIPS 2026 paper Senses Wide Shut: A Representation–Action Gap in Omnimodal LLMs. It tests whether an omnimodal model notices when a question makes a false claim about what the model sees or hears.
📄 Paper · 🌐 Project page · 💻 Code
videos/.| Standard premise (answer A–D) | Misleading premise (answer E or F) | |
|---|---|---|
| Vision | standard_vision (500) | misleading_vision (500) |
| Audio | standard_audio (500) | misleading_audio (500) |
A misleading question copies its standard twin and changes exactly one premise detail, such as a shirt colour or the kind of music that plays. Every question lists the same six options, where E is "The visual detail in the question is incorrect" and F is "The audio detail in the question is incorrect", so the right answer to a misleading question is to reject the premise. Each item also carries a question category, a 10-second answer timestamp and, for misleading items, the kind of detail that was changed. We checked all 2,000 items by hand while watching each clip with sound.
from datasets import load_dataset
ds = load_dataset("ngqtrung/IMAVB", split="misleading_audio")
row = ds[0]
print(row["question"], row["correct_answer"], row["video"]) # video path inside this repo
The annotations are released under CC BY-NC-SA 4.0. The clips come from public movie-scene channels (@BingeSociety, @BoxofficeMoviesScenes) and Condensed Movies. They remain the property of their copyright holders and are shared for non-commercial research and evaluation only. If you own a clip and want it removed, please open a discussion on this repository.
@inproceedings{nguyen2026senses,
title = {Senses Wide Shut: A Representation--Action Gap in Omnimodal {LLM}s},
author = {Nguyen Quang, Trung and Gao, Yiming and Pu, Fanyi and Zhang, Kaichen and Sun, Shuo and Liu, Ziwei},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
year = {2026}
}