LiveSports-3K Benchmark

News

[2025.05.12] We released the ASR transcripts for the CC track. See LiveSports-3K-CC.json for details.

Overview

LiveSports‑3K is a comprehensive benchmark for evaluating streaming video understanding capabilities of large language and multimodal models. It consists of two evaluation tracks:

• Closed Captions (CC) Track: Measures models’ ability to generate real‑time commentary aligned with the ground‑truth ASR transcripts.
• Question Answering (QA) Track: Tests models on multiple‑choice questions that probe semantic understanding.

The benchmark is introduced in the CVPR 2025 paper: LiveCC: Learning Video LLM with Streaming Speech Transcription at Scale. [Paper][Code]

Dataset Structure

├── LiveSports-3K-CC.json       # CC track (1,702 events)
└── LiveSports-3K-QA.json       # QA track (1,174 multiple-choice questions)

Closed Captions (CC) Track

• Total events: 1,702
• JSON file:
  • LiveSports-3K-CC.json: 1702 events, with previous and ground-truth ASR transcripts provided.

File format (*.json):

Question Answering (QA) Track

• Total questions: 1,174 four‑option MCQs
• JSON file:
  • LiveSports-3K-QA.json

File format (*.json):

Evaluation Protocols

CC Track

1. Input: Video title + previous CC (or empty if first event).
2. Task: Generate the next ASR caption for the event segment.
3. Metric: Win rate against GPT‑4o baseline, judged by GPT‑4o on stylistic and semantic alignment with ground truth.

QA Track

1. Input: Video clip frames + question + four answer options.
2. Task: Select the correct option.
3. Metric: Accuracy (percentage of correctly answered questions).

For distributed evaluation used in our paper, please refer to https://github.com/showlab/livecc.

Citation

@article{livecc,
  author       = {Joya Chen and Ziyun Zeng and Yiqi Lin and Wei Li and Zejun Ma and Mike Zheng Shou},
  title        = {LiveCC: Learning Video LLM with Streaming Speech Transcription at Scale},
  journal      = {arXiv preprint arXiv:2504.16030}
  year         = {2025},
}