ReactiveGWM-Datasets: Strategy-Aligned Rollouts for Reactive Game World Models

📚 Datasets-Introduction

ReactiveGWM-Datasets is the strategy-aligned training corpus that powers ReactiveGWM, a game world model that decouples player control from NPC autonomy. To learn that decoupling, the model needs supervision that pairs each gameplay clip with both a per-frame action stream (what the player did) and a high-level NPC description (what the NPC tried to do, and under which strategy).

This dataset provides exactly that, on two classic Capcom fighting games:

Each clip comes with two parallel annotations: a structured prompt that exposes NPC Active_Behavior, Passive_Behavior, and one of three strategy classes (Offense / Control / Defense) — used to train ReactiveGWM's cross-attention NPC modules — and a free-form vanilla narration prompt suitable for vanilla game model baselines.

🗂️ Directory layout

SF2/
├── clips/
│   ├── clip_0000000/
│   │   ├── video.mp4         # 5-second gameplay clip
│   │   └── actions.parquet   # per-frame controller inputs for both players
│   └── ...
├── metadata.csv              # video, action, structured NPC behavior + strategy
└── metadata_vanilla.csv      # video, action, natural-language narration
SF3/   (same layout)

metadata.csv — structured strategy prompt

Each row is video, action, prompt. The prompt column tags the NPC's moves and ends with one of the three strategy classes that ReactiveGWM's cross-attention modules condition on:

NPC: Active_Behavior(<move>: <desc>; ...),
     Passive_Behavior(<move>: <desc>; ...),
     Strategy(Offense | Control | Defense : <desc>)

Example (SF2):

NPC: Active_Behavior(Walk Right: Moves character towards the right side of the screen.), Passive_Behavior(Standing Block: Passively guards against high or mid attacks while standing.; Take Damage: ...; Knockback: ...; Knockdown: ...; Wake Up: ...), Strategy(Defense: Absorbs and evades incoming pressure, recovering safely instead of trading hits.)

metadata_vanilla.csv — natural-language prompt

Same video, action pairs but with free-form English narration, suitable for prompting vanilla text-to-video baselines:

"Street Fighter 2, At the Air Force Base, Ryu on the left crouches and then throws a standing punch that hits Guile on the right, after which Ryu walks right and throws another standing punch as Guile walks left and throws his own standing punch."

actions.parquet

Per-frame discrete controller state for both players (buttons + d-pad), aligned 1-to-1 with the video frames. This is the signal injected as a lightweight additive bias into the diffusion backbone in ReactiveGWM.

🛠️ How the data was generated

Rollouts are produced by a random agent fighting the built-in arcade AI, recorded through stable-retro (SF3 runs on a CPS-3 integration). The full pipeline — record → split into 5-second clips → multi-stage labelling (initial → class-balance → final fact-check → hard-rule strategy classification) → optional visual labelling — lives in a separate code release and is not distributed in this repository.

Annotations in the prompt columns are produced by an LLM (Gemini) over the recorded clips and then filtered with hard rules. They are machine-generated and may contain residual noise.

🤓 Acknowledgments

Gameplay recording uses the stable-retro framework (SF3 via a CPS-3 custom integration on top of FBNeo). NPC strategy annotations are produced by Gemini. The Street Fighter games themselves are © Capcom Co., Ltd.; see the IP notice below. We thank the open-source community for their valuable contributions.

🔗 Citation

If you use this dataset, please cite:

@misc{wang2026reactivegwmsteeringnpcreactive,
      title={ReactiveGWM: Steering NPC in Reactive Game World Models}, 
      author={Zeqing Wang and Danze Chen and Zhaohu Xing and Zizhao Tong and Yinhan Zhang and Xingyi Yang and Yeying Jin},
      year={2026},
      eprint={2605.15256},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2605.15256}, 
}

---

⚖️ Copyright, IP, and licensing

Read this before using the dataset.

This dataset contains derivative material from two commercial video games whose copyright the dataset authors do not own:

• Street Fighter II: Champion Edition — © Capcom Co., Ltd.
• Street Fighter III: New Generation / 3rd Strike — © Capcom Co., Ltd.

All in-game visuals, character designs, sprites, backgrounds, audio, and game mechanics depicted in the .mp4 clips remain the exclusive property of Capcom Co., Ltd. No ownership over the underlying games is claimed or transferred by this release.

What is and isn't licensed

The CC BY-NC 4.0 license declared in the YAML header applies only to the contributions that are original to this dataset:

• the text annotations in metadata.csv and metadata_vanilla.csv (the prompt column),
• the dataset's organisation, splits, and file naming,
• the per-frame actions.parquet action traces produced by the recording scripts.

The video frames and audio inside clips/*/video.mp4 are made available solely under fair-use / fair-dealing for non-commercial academic research on game AI, world models, and video understanding. They are not relicensed, and CC BY-NC 4.0 does not grant you any right to the underlying Capcom IP. See the LICENSE and NOTICE files shipped alongside this README for the full statement.

What you may NOT do

• Use the clips, or any model trained primarily on them, in a commercial product, including a game, a streaming service, or a paid API.
• Redistribute the clips bundled with game ROMs, BIOS files, or emulator binaries.
• Use the data to reproduce, recreate, or compete with Capcom's Street Fighter games or any other Capcom title.
• Claim ownership of, or relicense, the underlying game footage.

What you should do

• Treat this as an academic, non-commercial research artifact only.
• Cite both this dataset and acknowledge Capcom as the copyright holder of the source games in any publication or release that uses it.
• If you publish a model trained on this data, carry forward the same non-commercial restriction and copyright notice.

ROMs

No game ROMs are included in this repository, and none will be provided. Reproducing the dataset from the upstream generation code requires you to obtain legal copies of the original Street Fighter II (Genesis) and Street Fighter III (CPS3) ROMs yourself, in accordance with the laws of your jurisdiction.

Takedown / DMCA

If you are a rights holder (in particular Capcom Co., Ltd. or an authorised representative) and believe this dataset infringes your rights, please contact zeqing.wang@u.nus.edu with a description of the material in question. The maintainer will respond and, where appropriate, remove the contested content promptly.