A Benchmark and Agentic Framework for Omni-Modal Reasoning and Tool Use in Long Videos
See the code
*Equally contributing first authors
LongShOT introduces a diagnostic benchmark for long-form omni-modal video understanding. LongShOTBench features 3,401 intent-driven single- and multi-turn questions over 274 long-form videos (~41 min avg, ~188 hours total), probing visual, speech, and ambient-audio reasoning. Each sample includes reference answers and weighted criterion-level rubrics for interpretable evaluation with partial credit. LongShOTAgent is a training-free agentic baseline that couples full-video preprocessing with targeted retrieval, segment refinement, and claim verification over visual, speech, and audio evidence. Comprehensive evaluation of 105+ models reveals significant performance gaps, with the strongest closed-source API (Gemini 3.1 Pro) reaching 55.63%, the best open-source model (Qwen3-Omni 30B) reaching 64.05%, and LongShOTAgent achieving 66.64%.
Figure 1: LongShOTBench at a glance. An illustrated hour-long surprise party video shows how understanding a single scene requires fusing visual cues, speech, and non-speech audio scattered across temporally distant moments. LongShOTBench captures this challenge through intent-driven single- and multi-turn Q&A paired with weighted criterion-level rubrics, scored against a human-annotated ground truth.
Figure 2: Construction pipeline of LongShOTBench. Starting from raw long-form videos, speech, visual, and audio cues are extracted and fused into segment-wise aligned omni-modal metadata. The distilled metadata drives scenario and task mapping, followed by single- and multi-turn Q&A generation and verifiable rubric construction. Human validators review and correct the Q&A pairs and tailored rubrics, ensuring a grounded and reliable benchmark.
Create and activate a conda environment with Python 3.11:
conda create -n longshot python=3.11 -y
conda activate longshot
pip install -r requirements.txt
Note: If you encounter CUDNN issues, install it separately for your version:
conda install -c conda-forge cudnn=8
The preprocessing pipeline downloads videos from YouTube and processes them to extract video, audio, and speech metadata/transcriptions.
To download videos from YouTube, you need to provide authentication cookies:
preprocess/cookies.txtNavigate to the preprocess directory and download the Audio Flamingo model:
cd preprocess
hf download nvidia/audio-flamingo-3 --local-dir ./MODELS/audio-flamingo-3/
Download videos from YouTube:
python download_youtube_videos.py
Note: Some videos may fail to download through the script. Please manually download any missing videos and place them in the appropriate directory (preprocess/dataset/videos).
Start the VLM server in a separate terminal:
./vllm_start.sh vlm
In another terminal, run the VLM processing stages:
python run.py vlm-stages
Start the LLM server in a separate terminal:
./vllm_start.sh llm
In another terminal, run the LLM processing stages:
python run.py llm-stages
Once complete, the raw dataset with video, audio, and speech descriptions will be ready in the preprocess/dataset/ directory.
Generate the LongShOTBench dataset from the preprocessed videos:
cd ../datagen
python main.py
The final dataset will be saved at datagen/results/final_dataset.jsonl.
To share the dataset on Hugging Face Hub:
from datasets import Dataset
dataset = Dataset.from_json("results/clean_dataset.jsonl")
dataset.push_to_hub("your-org/longshot-bench", config_name="postvalid")
Evaluate model performance on the LongShOTBench dataset.
To generate responses from candidate models:
cd ../eval
bash generate.sh
To evaluate the generated responses:
bash eval.sh
Results will be saved in the eval/results_postvalid/ directory.
LongShOTAgent is a training-free omni-modal evidence-seeking agent that serves as a reference baseline for LongShOTBench. It preprocesses a video into a searchable multimodal store and operates a ReAct-style search–refine–verify loop at query time. See longshotagent/ for setup and usage instructions.
Figure 3: LongShOTAgent architecture. The indexer embeds video frames, speech transcripts and audio understanding into a per-video vector store. At query time, the orchestrator LLM analyzes multimodal evidence by issuing tool calls in a ReAct-style loop: search_video for semantic retrieval, refine_video for fine-grained segment-level re-analysis with specialists, and verify_claim for evidence grounding.
A short walkthrough of LongShOTAgent answering a query end-to-end — retrieving, refining, and verifying omni-modal evidence before producing a grounded answer.
https://github.com/user-attachments/assets/c6817e01-2ff2-44f1-a48a-cf174e4e41c2
LongShOTAgent can make mistakes. Please keep the following in mind:
This work is partially supported by the Meta Regional Research Grant, Project OMER, the Google Gift Research Award, and the NVIDIA Academic Grant.
If you find this work useful, please cite our paper:
@misc{kurpath2026benchmarkomnimodalreasoninglong,
title={A Benchmark for Omni-Modal Reasoning in Long Videos},
author={Mohammed Irfan Kurpath and Jaseel Muhammad Kaithakkodan and Jinxing Zhou and Sahal Shaji Mullappilly and Mohammad Almansoori and Noor Ahsan and Beknur Kalmakhanbet and Sambal Shikhar and Rishabh Lalla and Jean Lahoud and Mariette Awad and Fahad Shahbaz Khan and Salman Khan and Rao Muhammad Anwer and Hisham Cholakkal},
year={2026},
eprint={2512.16978},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2512.16978},
}
A Benchmark and Agentic Framework for Omni-Modal Reasoning and Tool Use in Long Videos
See the code
*Equally contributing first authors
LongShOT introduces a diagnostic benchmark for long-form omni-modal video understanding. LongShOTBench features 3,401 intent-driven single- and multi-turn questions over 274 long-form videos (~41 min avg, ~188 hours total), probing visual, speech, and ambient-audio reasoning. Each sample includes reference answers and weighted criterion-level rubrics for interpretable evaluation with partial credit. LongShOTAgent is a training-free agentic baseline that couples full-video preprocessing with targeted retrieval, segment refinement, and claim verification over visual, speech, and audio evidence. Comprehensive evaluation of 105+ models reveals significant performance gaps, with the strongest closed-source API (Gemini 3.1 Pro) reaching 55.63%, the best open-source model (Qwen3-Omni 30B) reaching 64.05%, and LongShOTAgent achieving 66.64%.
Figure 1: LongShOTBench at a glance. An illustrated hour-long surprise party video shows how understanding a single scene requires fusing visual cues, speech, and non-speech audio scattered across temporally distant moments. LongShOTBench captures this challenge through intent-driven single- and multi-turn Q&A paired with weighted criterion-level rubrics, scored against a human-annotated ground truth.
Figure 2: Construction pipeline of LongShOTBench. Starting from raw long-form videos, speech, visual, and audio cues are extracted and fused into segment-wise aligned omni-modal metadata. The distilled metadata drives scenario and task mapping, followed by single- and multi-turn Q&A generation and verifiable rubric construction. Human validators review and correct the Q&A pairs and tailored rubrics, ensuring a grounded and reliable benchmark.
Create and activate a conda environment with Python 3.11:
conda create -n longshot python=3.11 -y
conda activate longshot
pip install -r requirements.txt
Note: If you encounter CUDNN issues, install it separately for your version:
conda install -c conda-forge cudnn=8
The preprocessing pipeline downloads videos from YouTube and processes them to extract video, audio, and speech metadata/transcriptions.
To download videos from YouTube, you need to provide authentication cookies:
preprocess/cookies.txtNavigate to the preprocess directory and download the Audio Flamingo model:
cd preprocess
hf download nvidia/audio-flamingo-3 --local-dir ./MODELS/audio-flamingo-3/
Download videos from YouTube:
python download_youtube_videos.py
Note: Some videos may fail to download through the script. Please manually download any missing videos and place them in the appropriate directory (preprocess/dataset/videos).
Start the VLM server in a separate terminal:
./vllm_start.sh vlm
In another terminal, run the VLM processing stages:
python run.py vlm-stages
Start the LLM server in a separate terminal:
./vllm_start.sh llm
In another terminal, run the LLM processing stages:
python run.py llm-stages
Once complete, the raw dataset with video, audio, and speech descriptions will be ready in the preprocess/dataset/ directory.
Generate the LongShOTBench dataset from the preprocessed videos:
cd ../datagen
python main.py
The final dataset will be saved at datagen/results/final_dataset.jsonl.
To share the dataset on Hugging Face Hub:
from datasets import Dataset
dataset = Dataset.from_json("results/clean_dataset.jsonl")
dataset.push_to_hub("your-org/longshot-bench", config_name="postvalid")
Evaluate model performance on the LongShOTBench dataset.
To generate responses from candidate models:
cd ../eval
bash generate.sh
To evaluate the generated responses:
bash eval.sh
Results will be saved in the eval/results_postvalid/ directory.
LongShOTAgent is a training-free omni-modal evidence-seeking agent that serves as a reference baseline for LongShOTBench. It preprocesses a video into a searchable multimodal store and operates a ReAct-style search–refine–verify loop at query time. See longshotagent/ for setup and usage instructions.
Figure 3: LongShOTAgent architecture. The indexer embeds video frames, speech transcripts and audio understanding into a per-video vector store. At query time, the orchestrator LLM analyzes multimodal evidence by issuing tool calls in a ReAct-style loop: search_video for semantic retrieval, refine_video for fine-grained segment-level re-analysis with specialists, and verify_claim for evidence grounding.
A short walkthrough of LongShOTAgent answering a query end-to-end — retrieving, refining, and verifying omni-modal evidence before producing a grounded answer.
https://github.com/user-attachments/assets/c6817e01-2ff2-44f1-a48a-cf174e4e41c2
LongShOTAgent can make mistakes. Please keep the following in mind:
This work is partially supported by the Meta Regional Research Grant, Project OMER, the Google Gift Research Award, and the NVIDIA Academic Grant.
If you find this work useful, please cite our paper:
@misc{kurpath2026benchmarkomnimodalreasoninglong,
title={A Benchmark for Omni-Modal Reasoning in Long Videos},
author={Mohammed Irfan Kurpath and Jaseel Muhammad Kaithakkodan and Jinxing Zhou and Sahal Shaji Mullappilly and Mohammad Almansoori and Noor Ahsan and Beknur Kalmakhanbet and Sambal Shikhar and Rishabh Lalla and Jean Lahoud and Mariette Awad and Fahad Shahbaz Khan and Salman Khan and Rao Muhammad Anwer and Hisham Cholakkal},
year={2026},
eprint={2512.16978},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2512.16978},
}