TimeChat-Online-139K

Project Page | Paper | GitHub | Model Checkpoint

⚠️ Important Notice: Research-Only Use

Before downloading or using this dataset, you must agree to the LICENSE terms.

This dataset contains video data that may be copyright-sensitive. It is provided solely for non-commercial research and educational purposes. By accessing this dataset, you confirm that you understand and agree to the terms in the LICENSE file.

---

📦 Dataset Overview

For flexible real-time interaction, we introduce a comprehensive streaming video dataset TimeChat-Online-139K with backward-tracing, real-time visual perception, and future-responding scenarios.

• This repo contains .tar.gz archives of raw video frames extracted from diverse video sources. Each archive corresponds to a dataset.
• ⚠️ Raw video data may include copyrighted material; all use is governed by the LICENSE.
• 📋 We only release 1fps extracted video frames.

Statistics

• 11,043 visually informative videos (average duration: 11.1 minutes)
• 139K question-answer pairs covering backward tracing, real-time visual perception, and forward active responding
• Average of 87.8 scene-oriented key frames per video (~7.14 seconds between consecutive frames)

---

📋 Video Frame Caption Annotations

We now provide video frame caption annotations in annotations_caption_flt.jsonl. This file contains detailed frame-level captions for video key frames.

Format

Each line in the JSONL file represents a single frame annotation with the following structure:

{
  "frame_id": 142,
  "segment_id": 146, 
  "timestamp": 393.3,
  "caption": "This frame presents a stark contrast to the preceding one...",
  "video": "Youcook2/gZuDMKXWU_E"
}

Key Descriptions

• frame_id (int): Sequential key frame identifier within the video
• segment_id (int): Key segment identifier
• timestamp (float): Video timestamp in seconds of the key frame
• caption (str): Detailed textual description of the visual content in the frame via GPT-4o
• video (str): Video identifier in the format {dataset}/{video_id}

Coverage

The annotation file provides comprehensive coverage with 876,398 frame-level detailed captions across approximately 10,949 videos from multiple datasets. Each frame caption averages 176 words in length, with an average of 87.8 key frames per video. This extensive collection offers rich visual descriptions ideal for training and research in video understanding tasks.

---

🎯 Data Sources

The dataset consists of 11,043 videos sampled from the following 13 public video datasets:

Total videos: 11,043

Please refer to the original dataset papers and licenses:

• COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis.
• HD-VILA: Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions
• ViTT: Multimodal Pretraining for Dense Video Captioning
• VideoIC: VideoIC: A Video Interactive Comments Dataset and Multimodal Multitask Learning for Comments Generation
• QV-Highlights: Detecting Moments and Highlights in Videos via Natural Language Queries.
• YouCook2: Towards Automatic Learning of Procedures From Web Instructional Videos.
• QuerYD: A video dataset with high-quality text and audio narrations.
• Movie101: A New Movie Understanding Benchmark
• ActivityNet: A Large-Scale Video Benchmark for Human Activity Understanding
• TVSum: Summarizing web videos using titles
• YouMakeup: A Large-Scale Domain-Specific Multimodal Dataset for Fine-Grained Semantic Comprehension
• HiREST: Hierarchical Video-Moment Retrieval and Step-Captioning

---

🔒 License and Terms

This dataset is released under a custom research-only license. By accessing it, you agree:

• ❌ Not to use for any commercial purposes
• ❌ Not to redistribute or rehost
• ✅ To cite the dataset in your academic publications

For full terms, see LICENSE. Contact us if you're unsure about permitted uses.

---

📄 Citation

If you use this dataset in your research, please cite:

@misc{timechatonline,
    title={TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videos}, 
    author={Linli Yao and Yicheng Li and Yuancheng Wei and Lei Li and Shuhuai Ren and Yuanxin Liu and Kun Ouyang and Lean Wang and Shicheng Li and Sida Li and Lingpeng Kong and Qi Liu and Yuanxing Zhang and Xu Sun},
    year={2025},
    eprint={2504.17343},
    archivePrefix={arXiv},
    primaryClass={cs.CV},
    url={https://arxiv.org/abs/2504.17343},
}