Overview

This dataset provides a comprehensive collection for Online Spatial-Temporal Understanding tasks, covering multiple domains including Dense Video Captioning, Video Grounding, Step Localization, Spatial-Temporal Action Localization, and Object Tracking.

Data Formation

Our pipeline begins with 96K high-quality samples curated from 5 tasks across 12 datasets. The conversion process enhances online spatiotemporal understanding through template transformation. We strategically insert queries along the timeline in an organized interleaved format for each video sample to facilitate temporal context differentiation.

---

Additional Information:

• Interleave Format: Temporally Random Insert (T3, T2, T1)
• Video Timeline: Processed for Online Video LLM

Data Formats

• Format 1: Conversational QA (LLaVA-style)

{
    "video": "116/NLy71UrHElw.mp4",
    "conversations": [
        {
            "from": "human",
            "timestamps": 1026.0,  # Video timestamp in seconds
            "value": "<video>\nBased on current observation, list events..."
        },
        {
            "from": "gpt",
            "value": "21.0s - 22.0s (duration: 1.0s), begin to run up..."
        }
    ]
}

Format 2: Template-based Tracking

{
    "video": "GOT-10k_Train_006457",
    "fps": 1,  # Frame rate
    "all_image_files": ["00000001.jpg", ...],  # Keyframe paths
    "image_bboxes": [  # Temporal object tracking data
        {
            "timestamp": 0.0,
            "bbox": [0.412, 0.517, 0.452, 0.753]  # [x1,y1,x2,y2]
        },
        ...
    ],
    "query_template": {  # Randomized temporal insertion
        "from": "human",
        "value": "Track the location of \"person\" at <bbox> over time..."
    }
}

Source Data

Citation

If you find this project useful in your research, please consider cite:

@article{huang2024online,
  title={Online Video Understanding: A Comprehensive Benchmark and Memory-Augmented Method},
  author={Huang, Zhenpeng and Li, Xinhao and Li, Jiaqi and Wang, Jing and Zeng, Xiangyu and Liang, Cheng and Wu, Tao and Chen, Xi and Li, Liang and Wang, Limin},
  journal={arXiv preprint arXiv:2501.00584},
  year={2024}
}