Realtime-QA-100K

📄 Tech Report  |  💻 GitHub  

Realtime-QA-100K is a 100K-sample realtime video question answering dataset constructed from YouTube videos. Each sample contains a multimodal conversation and frame timestamp metadata that aligns every <|video|> token in the assistant text with one video frame timestamp.

This repository does not contain the actual video files. We only provide annotations, YouTube video IDs, and timestamp metadata. Users are responsible for downloading the videos themselves in compliance with YouTube's Terms of Service and the original uploaders' licenses.

Pipeline at a Glance

Figure 1. End-to-end construction pipeline of Realtime-QA-100K.

The construction of Realtime-QA-100K follows a highly structured, multi-layer data synthesis pipeline:

• Input Layer processes raw hierarchical video captions (covering video, event, and action-level annotations) and temporally aligns them into a single, flattened Caption Timeline of structured observations \({t_i, c_i}\).
• Stage A — Semantic Construction — What Changed → What to Ask → How to Answer — handles the semantic logic: it identifies key transition points, drafts state-dependent user instructions, and generates chronological state-aware responses (or silent decisions) without future-information leakage.
• Stage B — Temporal Layout — When to Ask → Where Tokens Go → Stream Assembly — maps these semantic decisions onto a second-by-second physical timeline: it schedules temporal anchors, simulates dynamic token emission rates, and interleaves <|video|>, text, and <|silence|> slots.
• Output Layer performs multi-turn dialogue assembly, stitching interleaved streams into coherent, multi-turn conversations (including preemption scenarios) to produce the final training_sample.json represented in the messages field.

Dataset Summary

• Samples: 100,000 (uniform random subsample, seed=42)
• Source corpus: 272,910 realtime QA rows; 128,967 are YouTube-only
• Source platform: YouTube
• Unique YouTube videos: 24,817
• Duration bucket: within 4 minutes
• Frame sampling: 1 fps
• Languages: English (93,815), Chinese (6,185)
• Categories: 27
• <|video|> / frame_timestamps alignment failures in release build: 0
• License for annotations: CC-BY-NC-4.0

Data Provenance

The 100K release is a deterministic subsample of a larger realtime video QA training JSONL used for streaming video LLM fine-tuning. Only rows whose video source is YouTube are eligible; the final set is drawn uniformly at random with seed=42.

Per-row fields are compacted for Hub distribution:

• The shared system prompt is stored once in system_prompt.txt instead of being repeated in every row.
• Platform-wide constants (YouTube URL template, 1 fps sampling, duration bucket) are documented here rather than duplicated per sample.
• Legacy [RealTime QA][INFO] / [RealTime QA][PLACEHOLDER] markers from the source format are removed; their information is represented by video.video_id, video.category, video.subcategory, and video.frame_timestamps.

YouTube URL template: https://www.youtube.com/watch?v={video_id}

Dataset Structure

The dataset has one split, train, stored as Parquet shards:

data/train-00000-of-00002.parquet
data/train-00001-of-00002.parquet

Each row has the following schema:

{
  "id": "rtqa_000000",
  "language": "en",
  "messages": [
    {"role": "user", "content": ""},
    {"role": "assistant", "content": "<|silence|><|video|>..."},
    {"role": "user", "content": "..."},
    {"role": "assistant", "content": "...<|video|>..."}
  ],
  "video": {
    "video_id": "hWYS_j90p-I",
    "category": "Art",
    "subcategory": "ugee_tablet",
    "frame_timestamps": ["00:00", "00:01"]
  }
}

Global metadata files:

• system_prompt.txt: shared system prompt removed from individual rows.
• video_ids.txt: unique YouTube video IDs in the 100K sample.

Integrity invariant. For every row, the number of <|video|> tokens appearing in all assistant messages equals len(video.frame_timestamps). This was checked on the full 100K build with zero mismatches.

Special Tokens

• <|video|>: marks the position corresponding to a video frame. The number of <|video|> tokens in the assistant messages equals len(video.frame_timestamps).
• <|silence|>: marks a timestep where the assistant should remain silent.
• <|...|>: turn-break token. Because the dataset is built for real-time streaming, an assistant answer may still be unfolding when the video moves on to a new scene. In that case the model needs to wrap up the current reply quickly and yield so the next turn can respond to the new scene. <|...|> marks exactly that intentional early cut-off — it is a designed token in the conversation, not a text truncation or an ellipsis.

Loading

Install

pip install -U datasets pyarrow huggingface_hub

Load the train split

from datasets import load_dataset

ds = load_dataset("OpenMOSS-Team/Realtime-QA-100K", split="train")
print(ds)
# Dataset({features: ['id', 'language', 'messages', 'video'], num_rows: 100000})
print(ds[0])

This downloads the two Parquet shards (about 95 MB in total) into the Hugging Face cache and returns a Dataset with four fields: id, language, messages, and video.

Streaming mode (no full download)

from datasets import load_dataset

ds = load_dataset("OpenMOSS-Team/Realtime-QA-100K", split="train", streaming=True)
for ex in ds.take(3):
    print(ex["id"], ex["language"], len(ex["video"]["frame_timestamps"]))

Filter by language or category

ds = load_dataset("OpenMOSS-Team/Realtime-QA-100K", split="train")
en_ds = ds.filter(lambda x: x["language"] == "en")
art_ds = ds.filter(lambda x: x["video"]["category"] == "Art")
print(len(en_ds), len(art_ds))

Rebuild the full chat with the shared system prompt

Every row omits the system message because every example uses the same prompt, stored in system_prompt.txt. Download it once, then prepend it to each row:

from datasets import load_dataset
from huggingface_hub import hf_hub_download

ds = load_dataset("OpenMOSS-Team/Realtime-QA-100K", split="train")
system_prompt_path = hf_hub_download(
    repo_id="OpenMOSS-Team/Realtime-QA-100K",
    filename="system_prompt.txt",
    repo_type="dataset",
)
with open(system_prompt_path, "r", encoding="utf-8") as f:
    system_prompt = f.read()

def to_chat(example):
    return [{"role": "system", "content": system_prompt}] + example["messages"]

chat = to_chat(ds[0])
for msg in chat[:3]:
    print(msg["role"], "->", msg["content"][:80])

Recover the YouTube URL

example = ds[0]
url = f"https://www.youtube.com/watch?v={example['video']['video_id']}"
print(url)

Read Parquet shards directly

If you prefer raw Parquet I/O without the datasets package:

import pyarrow.parquet as pq
from huggingface_hub import hf_hub_download

shard_path = hf_hub_download(
    repo_id="OpenMOSS-Team/Realtime-QA-100K",
    filename="data/train-00000-of-00002.parquet",
    repo_type="dataset",
)
table = pq.read_table(shard_path)
print(table.schema)
print(table.num_rows)

Or with pandas:

import pandas as pd

df = pd.read_parquet(shard_path)
print(df.head())

Video Access

Video files are not included. Each example provides:

• video.video_id: YouTube video ID.
• video.frame_timestamps: timestamps sampled at 1 fps.
• video.category and video.subcategory: original topic labels from data collection.

Users should independently obtain videos and align frames using video.frame_timestamps.

Statistics

Language Distribution

Message Lengths

The shared system prompt is stored separately in system_prompt.txt. Each row therefore contains either 4 messages (one user–assistant turn) or 6 messages (two turns).

Frame Timestamp Lengths

len(video.frame_timestamps) per sample at 1 fps (one timestamp per second of video coverage in the conversation).

The pie chart below groups lengths into 1-minute buckets; summary stats also appear in the figure legend.

Category Distribution

License and Usage

The annotation data in this repository is released under CC-BY-NC-4.0.

Copyright Notice

This repository does not host or redistribute any raw video files. All video IDs and metadata belong to the respective content creators on YouTube. The copyright of the original video content remains entirely with the original owners/uploaders. This dataset only provides annotations, indices, and metadata for academic and research purposes.

Compliance Disclaimer

Users are solely responsible for obtaining the video files independently. Any downloading or processing of YouTube videos must be done in strict compliance with YouTube's Terms of Service and the licenses selected by the original uploaders. The dataset creators are not liable for any terms of service violations or copyright infringements arising from users' independent data acquisition.

This dataset is intended for academic research on real-time and streaming video understanding. It should not be used for commercial deployment, face recognition, surveillance, or other applications that may violate privacy or platform terms.

Limitations

• Video files are not directly included, meaning exact reproducibility depends on the availability of the original YouTube videos (which may become unavailable over time due to deletion, privacy, or regional restrictions); additionally, comprehensive PII, NSFW, and sensitive-content audits have not yet been performed.
• The granularity of the real-time data synthesis can be further refined to produce more natural and seamless results.

Citation

@article{wang2026mossvideo,
  title         = {{MOSS-Video-Preview: Toward Real-Time Video Understanding via Cross-Attention}},
  author        = {Pengyu Wang, Chenkun Tan, Shaojun Zhou, Wei Huang, Qirui Zhou, Zhan Huang, Zhen Ye, Jijun Cheng, Xiaomeng Qian, Yanxin Chen, Xingyang He, Huazheng Zeng, Chenghao Wang, Pengfei Wang, Hongkai Wang, Shanqing Gao, Yixian Tian, Chenghao Liu, Xinghao Wang, Botian Jiang, Xipeng Qiu},
  year          = {2026},
  journal       = {arXiv preprint arXiv:2606.07639},
  eprint        = {2606.07639},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2606.07639}
}