---

About this list

This repository tracks action-conditioned visual world models (ACWMs) across three connected use cases:

The list includes models that generate future pixels/video as well as models that roll the world forward in a learned visual latent space. Last metadata check: 2026-08-14.

Contents

• About this list
• Scope
• Tag legend
• Surveys and perspectives
• Game world models
• Embodied simulation
• Latent visual world models and control
• Benchmarks
• Datasets
• Related lists

Scope

We use action-conditioned world model (ACWM) for a model that learns controlled visual dynamics:

future visual states · rewards · termination
= f(observation history, actions, context)

Here, future states may be pixels, video tokens, object/3D states, or learned visual features; actions may be observed controls or learned latent actions; and context may include language, goals, scene information, or embodiment details.

Included work should satisfy these criteria:

1. Visual observations are modeled in pixel space, a renderable 3D/4D representation, discrete visual tokens, or a learned visual latent space.
2. Actions or controls affect the transition model. Keyboard/mouse inputs, robot commands, trajectories, action chunks, camera controls, and learned latent actions all qualify.
3. The learned transition is used for interactive rollout, simulation, planning, policy learning, evaluation, or control.

The sections are navigation aids rather than mutually exclusive theoretical categories. In particular, embodied papers are kept in one flat list and may carry several use tags.

Tag legend

Representation

• Video — future RGB/RGB-D or multi-view observations.
• Token — discrete visual tokens or autoregressive observation tokens.
• Latent — recurrent or transformer latent states learned from visual observations.
• Feature — dynamics in a pretrained visual feature space.
• 3D/4D — occupancy, point, Gaussian, flow, or other renderable spatial states.
• Unified — visual dynamics and action generation/decoding learned in one jointly trained model.

Downstream use

• Game — game-playing or game-like interactive environments.
• Interactive — controllable, multi-step visual rollouts.
• Eval — policy evaluation, ranking, verification, or failure detection.
• IL — imitation learning, behavior cloning, DAgger, or demonstration replay.
• RL — reinforcement learning or reinforcement fine-tuning inside the model.
• Plan — MPC, tree search, visual foresight, or goal-conditioned planning.
• Policy — joint world/action learning or direct policy improvement.
• Data — synthetic trajectories, observations, or demonstrations.

Surveys and perspectives

• “From World Models to World Action Models: A Concise Tutorial for Robotics,” arXiv 2026.07. [Paper]
• “Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends,” arXiv 2026.06. [Paper]
• “World Model for Robot Learning: A Comprehensive Survey,” arXiv 2026.05. [Paper] [Project] [List]
• “Video Generation Models as World Models: Efficient Paradigms, Architectures and Algorithms,” arXiv 2026.03. [Paper]
• “A Survey: Learning Embodied Intelligence from Physical Simulators and World Models,” arXiv 2025.07. [Paper] [List]

Game World Models

🎮 Game-like interactive environments. The learned model itself is a controllable game or game-like visual environment. Entries use the same table format as the embodied section and are ordered by first public release, newest first.

Embodied Simulation

🤖 Robot-facing world models. This is deliberately one flat list spanning video, latent, 3D/4D, and unified world models. The Uses column allows a paper to be simultaneously classified as evaluation, imitation learning, RL, planning, policy learning, and/or data generation.

Latent Visual World Models and Control

🧠 Latent imagination and visual control. This section covers general visual control and model-based RL work whose action-conditioned dynamics live partly or entirely in learned latent space. Pixel-generative methods are included when they are foundational to this line.

Benchmarks

Datasets

• DROID: Large-scale, in-the-wild robot manipulation trajectories. [Project] [Code]
• Open X-Embodiment: Cross-embodiment robot trajectories and RT-X models. [Project]
• BridgeData V2: Broad manipulation data collected across many environments and tasks. [Project]
• RoboNet: Multi-robot video data for learning visual dynamics. [Project]
• BAIR Robot Pushing: Action-conditioned pushing videos used by early visual-foresight work. [Dataset]

Related lists

• Awesome World Models for Robotics — structural inspiration and a broader world-model list.
• Awesome World Models — general world-model resources.
• Awesome World Model for Robotics Policy — robot-policy-oriented world models.
• Embodied World Models Survey — physical simulators and world models for embodied intelligence.
• Awesome Video Diffusion — broader video-diffusion literature.

License

This list is released under CC0 1.0.