GameplayQA: A Decision-Dense POV-Synced Multi-Video
Understanding Benchmark of 3D Virtual Agents

Overview

GameplayQA is the first benchmark for POV-Synced Multi-Video Understanding and Multi-Agent Video Understanding, built from ego-centric gameplay footage across 9 commercial 3D games. It features 2.4K multiple-choice questions spanning three levels of reasoning complexity — from single-clip action recognition to synchronized cross-video understanding.

Why synchronized multi-viewpoint reasoning matters

The ability to reason across multiple synchronized viewpoints is critical in many real-world domains: sports analytics leveraging multiple camera angles, autonomous driving requiring sensor fusion from surround cameras, law enforcement reviewing multiple dashcam feeds, and coordinated robot or drone fleets operating in shared environments. In esports and gaming, cross-POV synchronization and collective reasoning are fundamental to interpreting multi-agent collaboration — making gameplay an ideal controlled testbed for developing and evaluating these capabilities in video-language models.

Abstract

Multimodal LLMs are increasingly deployed as perceptual backbones for autonomous agents in 3D environments, from robotics to virtual worlds. These applications require agents to perceive rapid state changes, attribute actions to the correct entities, and reason about concurrent multi-agent behaviors from a first-person perspective — capabilities that existing benchmarks do not adequately evaluate. We introduce GameplayQA, a framework for evaluating agentic-centric perception and reasoning through video understanding. Specifically, we densely annotate multiplayer 3D gameplay videos at 1.22 labels/second, with time-synced, concurrent captions of states, actions, and events structured around a triadic system of Self, Other Agents, and the World — a natural decomposition for multi-agent environments. From these annotations, we generate 2.4K diagnostic QA pairs organized into three levels of cognitive complexity, accompanied by a structured distractor taxonomy that enables fine-grained analysis of where models hallucinate. Evaluation of frontier MLLMs reveals a substantial gap from human performance, with common failures in temporal and cross-video grounding, agent-role attribution, and handling the decision density of the game. We hope GameplayQA stimulates future research at the intersection of embodied AI, agentic perception, and world modeling.

Dataset Files

Raw annotation files and video clips are located in the annotation/ folder, organized by project batch (e.g. annotation/project-batch1/, annotation/project-multi-batch1/). Processed benchmark files ready for evaluation are provided as CSV files at the root level.

Download/Loading the Dataset

The QA metadata (CSV files) can be loaded directly via the HuggingFace datasets library:

from datasets import load_dataset

# Main benchmark (default)
ds = load_dataset("wangyz1999/GameplayQA")

# Generalization benchmark
ds = load_dataset("wangyz1999/GameplayQA", "generalization")

The raw video clips are stored under annotation/ and tracked with Git LFS. To download them, clone the full repository:

git lfs install
git clone https://huggingface.co/datasets/wangyz1999/GameplayQA

Each QA row contains video_start / video_end timestamps in seconds. Evaluation typically requires cropping the referenced clip to the relevant segment before passing it to a model.

Column Reference

Column	Type	Description
id	string	Unique question ID (e.g. q-1767607156491-x4371a218)
instance_id	string	Video clip instance ID (e.g. i-052)
game_name	string	Game title. One of: ApexLegends, ARCRaiders, Battlefield6, CounterStrike2, Cyberpunk2077, EldenRing, Minecraft, NoManSky, Valheim
video_1_file_name … video_5_file_name	string | null	Paths to the video clip(s) (under annotation/). Single-video questions populate only video_1_file_name; multi-video questions populate up to 5 columns.
video_indices	string	0-based index/indices of the video(s) used, comma-separated (e.g. 0 or 0,1,2)
video_start	int	Start time of the relevant video segment (seconds)
video_end	int	End time of the relevant video segment (seconds)
question_code	string	Fine-grained question type code (e.g. OA-IDENT, V1-SA2V2-SA-IDENT)
task_name	string	Human-readable task category derived from question_code. See Task Categories below.
question_level	int	Context scope level: 1 = Single Reference, 2 = Temporal, 3 = Cross-Video
question	string	The question text
correct_option	string	The correct answer text
answer_start	int | null	Start time of the answer evidence segment (seconds). Null for existence/binary questions.
answer_end	int | null	End time of the answer evidence segment (seconds). Null for existence/binary questions.
distractor_1 … distractor_3	string | null	Wrong answer options (up to 3)
distractor_1_type … distractor_3_type	string | null	Distractor source. One of: lexical, temporal, role, scene, binary, count, order, intent, cross-video

Taxonomy

Entity Types: Self — Other — World

GameplayQA organizes perception in interactive 3D environments around a tripartite entity decomposition that mirrors how agents must reason in multi-agent settings.

Cognitive Levels

Questions are organized by the amount of temporal and cross-video context required:

Task Categories

Questions are organized into 15 task categories across 3 context scope levels (2,365 total questions).

Source Data

GameplayQA is built from first-person gameplay footage sourced from 9 commercially released multiplayer games spanning diverse genres:

• Single-POV games: Minecraft, Apex Legends, No Man's Sky, Elden Ring, Cyberpunk 2077, Valheim
• Multi-POV synchronized games: Counter-Strike 2, Battlefield 6, ARC Raiders

Videos were sourced from YouTube, Twitch streams, and existing datasets (Counter-Strike 2 footage from X-EGO-CS). For multi-POV games, groups of streamers who played together in the same match were identified and their individual recordings were manually time-aligned to construct temporally synchronized multi-video sets.

Question Generation

Questions are generated through a combinatorial template-based algorithm that systematically combines verified annotation labels across five orthogonal dimensions: number of videos (single/multi), context target (summative/timestamp/entity/cross-video referring), entity type (SA/SS/OA/OS/WO/WE), distractor type, and question form. The algorithm initially produces ~400K candidate QA pairs; strategic downsampling to 4K enforces balanced category coverage before quality assurance yields the final 2,365 gold-standard pairs.

Distractor Types

Type	Description
lexical	Text-based variants of the correct answer (synonyms, antonyms, attribute changes)
temporal	Events that did occur, but outside the queried time window
role	Correct event but attributed to the wrong agent
scene	Plausible events that never occurred in the video
cross-video	Events from a different synchronized video perspective
binary	Negation of a binary (true/false) answer
count	Incorrect quantity for counting questions
order	Incorrect temporal ordering of events
intent	Alternative plausible motivations for an action

Quality Assurance

Language prior filtering: Each question is queried with text only (no video) using Gemini Flash with k=3 trials. Questions where the model consistently selects the correct answer without visual grounding are removed to prevent exploitation of statistical regularities in question phrasing.

Human evaluation: A stratified sample of 120 questions covering all question types was reviewed by annotators, who verified that (1) the video contains exactly one unambiguous correct answer, and (2) the question adheres to the semantics of its question code. Questions flagged as faulty (~8%) were corrected or removed.

Annotations

Annotation Process

Videos were annotated using dense multi-track timeline captioning via a custom-built annotation tool — sync-video-label. Each of the six entity types (SA, SS, OA, OS, WO, WE) is treated as an independent annotation track, and labels within and across tracks can overlap temporally to capture concurrent events.

The process follows a two-stage human-in-the-loop workflow:

1. Stage 1 — AI-assisted generation + human verification: Gemini Pro generates candidate labels and distractors (3,632 predictions). Four graduate student annotators then verify and refine: 31.1% of predicted labels were deleted, 42.7% were edited (61.9% requiring caption changes, 42.2% requiring temporal boundary adjustments), and 7.6% of final labels were added manually.
2. Stage 2 — Independent review: A separate annotator reviews all labels, making further adjustments to ~12% of labels.

A live read-only demo of the annotation interface is available at sync-video-label.vercel.app. See the demo video for a walkthrough.

Annotators

The annotation team consisted of 5 graduate students (ages 21–31). All annotators were experienced gamers: 60% play 3–5 times per week, 60% have 8+ years of gaming experience. Roles were distributed as 4 labelers and 2 evaluators, with one participant serving in both capacities for cross-stage consistency.

Label Statistics

A total of 2,709 true labels were annotated across 2,219 seconds of footage, yielding a decision density of ρ ≈ 1.22 labels/second — roughly one decision-relevant event per second.

Related Datasets

• wangyz1999/X-EGO-CS — Cross-ego synchronized gameplay data from Counter-Strike 2, used as the source for the CounterStrike2 split in this benchmark.

Citation

If our research is helpful to you, please cite our paper:

@article{wang2026gameplayqa,
  title   = {GameplayQA: A Benchmarking Framework for Decision-Dense POV-Synced Multi-Video Understanding of 3D Virtual Agents},
  author  = {Wang, Yunzhe and Xu, Runhui and Zheng, Kexin and Zhang, Tianyi and Kogundi, Jayavibhav Niranjan and Hans, Soham and Ustun, Volkan},
  year    = {2026},
  eprint  = {2603.24329},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url     = {https://arxiv.org/abs/2603.24329}
}