VStream-QA Dataset Card

Dataset details

We proposed VStream-QA, a novel question answering benchmark specifically designed for online video streaming understanding. VS-Ego and VS-Movie are offline evaluation benchmarks, while RVS-Ego and RVS-Movie are online evaluation benchmarks.

Here are the statistics of the dataset:

Preparation

We build this test dataset on Ego4d and MovieNet.

1. Download videos. Please follow the instruction of their official website to download Ego4d video data and MovieNet keyframe data.

2. Cut videos. Please cut original videos according to following json files and save your video/image/feature files for each subset.

VStream-QA
├──vstream
│   ├──vs_ego.json
│   ├──vs_movie.json
├──vstream-realtime
│   ├──rvs_ego.json
│   ├──rvs_movie.json

For example, you need to cut the Ego4d video original_video.mp4 from start_time second to end_time second, and save the video clip as video_id.mp4. Note that for Movienet videos, you need to cut the keyframes from start_shot to end_shot.

    {
        "video_id": "000246",
        "original_video": "e58207f1-84ec-424b-a997-ff64a57eb13b",
        "start_time": 2430,
        "end_time": 3270
    }

2. Extract features(optional). We recommend to extract ViT features of all video clips and save them as video_id.safetensors with shape=[Length, 256, 1024] at fps=1. You can evaluate from video/image files directly but it will be slower.

Evaluation

Please follow the scripts in Flash-VStream to evaluate your model.

VS-Ego and VS-Movie benchmark is designed for offline long video question answer. Please use following annotations to evaluate your model:

VStream-QA
├──vstream
│   ├──test_qa_ego4d.json
│   ├──test_qa_movienet.json

Each item in the test_qa json file contains:

    {
        "id": "001464",  # The question id
        "video_id": "000246",  # The video id
        "question": "Was I at the office before playing with the Jenga blocks?",
        "answer": "Yes.",
        "answer_type": "Order Judging(Y/N)"  # The answer type of the question
        "duration": 841,  # The duration of the video clip (in seconds or keyframes)
    }

RVS-Ego and RVS-Movie benchmark is designed for online video streaming question answer. Please use following annotations to evaluate your model:

VStream-QA
├──vstream-realtime
│   ├──test_qa_ego4d.json
│   ├──test_qa_movienet.json

Each item in the test_qa json file contains:

    {
        "id": "001464",  # The question id
        "video_id": "000098",  # The video id
        "question": "Was I at the office before playing with the Jenga blocks?",
        "answer": "Yes.",
        "answer_type": "Order Judging(Y/N)",  # The answer type of the question
        "start_time": 2430,  # The start time of the ground truth interval (in second or keyframe id)
        "end_time": 3270,  # The end time of the ground truth interval (in second or keyframe id)
        "gt_duration": 841,  # The duration of the ground truth interval (in seconds or keyframes)
        "duration": 3270  # The duration of the video clip (in seconds or keyframes)
    }

License

This project is licensed under the CC-BY-NC-4.0 License.