LV-Haystack: Temporal Search for Long-Form Video Understanding

Jinhui Ye¹*,  Zihan Wang²*,  Haosen Sun²,  Keshigeyan Chandrasegaran¹, 
Zane Durante¹,  Cristobal Eyzaguirre¹,  Yonatan Bisk³,  Juan Carlos Niebles¹,  Ehsan Adeli¹, 
Li Fei-Fei¹,  Jiajun Wu¹,  Manling Li²
 Stanford University¹, Northwestern University², Carnegie Mellon University³
CVPR 2025 . Dataset is part of the T* project
🌎Website | 🧑‍💻Code | 📄arXiv | 🏆 Leaderboard (Coming Soon)

Dataset Sample

{
    'video_id': 'b6ae365a-dd70-42c4-90d6-e0351778d991',
    'vclip_id': '6338b73e-393f-4d37-b278-68703b45908c',
    'question_id': 10,
    'question': 'What nail did I pull out?',
    'answer': 'E',
    'frame_indexes_vclip': [5036, 5232], # the keyframe indexes in the vclip
    'frame_indexes_video': [5036, 5232], # the keyframe indexes in the video
    'choices': {
        'A': 'The nail from the front wheel fender',
        'B': 'The nail from the motorcycle battery compartment',
        'C': 'The nail from the left side of the motorcycle seat',
        'D': 'The nail from the rearview mirror mount',
        'E': 'The nail on the right side of the motorcycle exhaust pipe'
    },
    'video_metadata': {
        'CLIP-reference-interval-vclip': [180.0, 240.0],  # Time interval of the "vclip" that is considered to be important by CLIP. this is calculated by (CLIP-reference-interval-video - vclip-interval-in-video[0])
        'CLIP-reference-interval-video': [180.0, 240.0],  # Time interval of the "video" that is considered to be important by CLIP. This is originally from the **Ego4D dataset**, used in our work for annotators to quickly locate in the video.
        'vclip_interval_in_video': [0.0, 480.06667277018227], # the vclip start and end second, i.e., for [a, b], the vclip starts at the a second of the video, ends at the b second of the video
        'frame_count': 14155,  # Total number of frames in the video
        'frame_rate': 30.0,  # Frame rate of the video
        'duration': 471.8333435058594,  # Duration of the video that are valid and unbroken, in seconds
        'resolution': '454x256',  # Original resolution of the video
        'frame_dimensions': None,  # Frame dimensions (if available)
        'codec': 'N/A',  # Codec used for the video (if available)
        'bitrate': 0,  # Bitrate of the video (if available)
        'frame_dimensions_resized': [340, 256],  # Resized frame dimensions
        'resolution_resized': '340x256',  # Resized resolution
        'video_id': 'b6ae365a-dd70-42c4-90d6-e0351778d991'  # Unique video identifier
    }
}

Dataset exploration

add hyperlink to demo

Dataset Usage

from datasets import load_dataset
dataset = load_dataset("LVHaystack/LongVideoHaystack")
print(dataset['train'])

Dataset({
    features: ['video_id', 'vclip_id', 'question', 'options', 'answer', 'frame_indexes_video', 'frame_indexes_vclip', 'video_metadata'],
    num_rows: 12892
})

Download and Process Video Source

TODO: We plan to provide a script of how to download a subset from Ego4d and process them. Below is download part adapted from their official guide and we will add video2clip script soon.

pip install ego4d

ego4d --output_directory=your_path/videos/ \
  --datasets full_scale annotations \
  --metadata \
  --video_uid_file video_uids.txt

# python process_videos_to_clips.py # TODO

Please find video_uids.txt in our repo, or you can generate it by:

import datasets
metadata = datasets.load_dataset("LVHaystack/LongVideoHaystack-metadata")["metadata"]
with open("video_uids.txt", "w") as file:
    for video_id in list(set(metadata['video_id'])):
        file.write(video_id + " ")

To follow evaluation for LongVideoBench in our paper, please find script to transform LongVideoBench to LongVideoHaystack format in transform_longvideobench.py.

Dataset Statistics Summary

Evaluation scripts

Please refer to ./eval.py.

Contact

• Jinhui Ye: jinhuiyes@gmail.com
• Zihan Wang: zihanw@u.northwestern.edu (datasets)
• Haosen Sun: haosensun2026@u.northwestern.edu
• Keshigeyan Chandrasegaran: keshik@stanford.edu
• Anabella Aisaro: anabellaisaro2025@u.northwestern.edu
• Manling Li: manling.li@northwestern.edu

Citation

@misc{tstar,
      title={Re-thinking Temporal Search for Long-Form Video Understanding}, 
      author={Jinhui Ye and Zihan Wang and Haosen Sun and Keshigeyan Chandrasegaran and Zane Durante and Cristobal Eyzaguirre and Yonatan Bisk and Juan Carlos Niebles and Ehsan Adeli and Li Fei-Fei and Jiajun Wu and Manling Li},
      year={2025},
      eprint={2504.02259},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2504.02259}, 
}

Website template borrowed from HourVideo.