AIWorldLab/Awesome-Vision-World-Model

From Seeing to Knowing the World: A Survey of Vision World Models

193

38 commits

updated Aug 18, 2026

AIWorldLab.github.io/survey

See the code

README

Awesome Vision World Models

This repository accompanies our survey From Seeing to Knowing the World: A Survey of Vision World Models and maintains a structured collection of Vision World Model resources:

It organizes papers across 7 Vision World Model designs and 5 dataset/benchmark categories, with their corresponding arXiv IDs, GitHub repositories, and project pages.
It also collects broader resources for world modeling, including theoretical analyses, top-tier conference workshops, interesting repositories, downstream-task applications, and other useful perspectives.

For more details, kindly refer to our paper :rocket:

:newspaper: News

2026-07-07: Add 27 papers, including 19 world model methods and 8 datasets/benchmarks. Update the papers in the teaser figure through 2026-07-01.
2026-05-23: Add 16 papers, 3 github repositories and 5 workshops.

More updates

2026-05-18: Fix metadata for 2 papers.
2026-04-08: Add 45 papers.

1. Vision World Model Designs
2. Datasets & Benchmarks
- 2.1. Foundational World Modeling
  - General World Prediction and Simulation
  - Physics and Causality Benchmark
- 2.2. Domain-specific World Modeling
3. Others
4. Acknowledgements
Citations

1. Designs

1.1. Sequential Generation

Visual Autoregressive Modeling

Design	Paper	Link

`VideoWorld 2`	VideoWorld 2: Learning Transferable Knowledge from Real-world Videos
`iMoWM`	iMoWM: Taming Interactive Multi-Modal World Model for Robotic Manipulation
`PWM`	From Forecasting to Planning: Policy World Model for Collaborative State-Action Prediction
`SAMPO`	SAMPO: Scale-wise Autoregression with Motion Prompt for Generative World Models
`RynnVLA-001`	RynnVLA-001: Using Human Demonstrations to Improve Robot Manipulation
`OccTENS`	OccTENS: 3D Occupancy World Model via Temporal Next-Scale Prediction
`Genie 3`	Genie 3: A new frontier for world models
`I²-world`	I²-World: Intra-Inter Tokenization for Efficient Dynamic 4D Scene Forecasting
`UniVLA`	Unified Vision-Language-Action Model
`WorldVLA`	WorldVLA: Towards Autoregressive Action World Model
`RoboScape`	RoboScape: Physics-informed Embodied World Model
`Xray2Xray`	Xray2Xray: World Model from Chest X-rays with Volumetric Context
`RLVR-World`	RLVR-World: Training World Models with Reinforcement Learning
`MineWorld`	MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft
`UVA`	Unified Video Action Model
`SurgWM`	Surgical Vision World Model
`DWS`	Pre-Trained Video Generative Models as World Simulators
`VideoWorld`	VideoWorld: Exploring Knowledge Learning from Unlabeled Videos
`Drivingworld`	Drivingworld: Constructing world model for autonomous driving via video GPT
`Moto`	Moto: Latent motion token as the bridging language for learning robot manipulation from video
`WHALE`	WHALE: Towards Generalizable and Scalable World Models for Embodied Decision-making
`GR-2`	GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation
`LatentDriver`	Learning Multiple Probabilistic Decisions from Latent World Model in Autonomous Driving
`Renderworld`	Renderworld: World model with self-supervised 3D label
`VidIT`	Video In-context Learning: Autoregressive Transformers are Zero-Shot Video Imitators
`iVideoGPT`	iVideoGPT: Interactive VideoGPTs are Scalable World Models
`Genie`	Genie: Generative Interactive Environments
`LWM`	World Model on Million-Length Video And Language With Blockwise RingAttention
`WHAM`	World and human action models towards gameplay ideation
`WorldDreamer`	WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokens
`OccWorld`	OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving
`GAIA-1`	Gaia-1: A generative world model for autonomous driving

MLLM-guided Multimodal Autoregressive Model

Design	Paper	Link

`WLA`	World-Language-Action Model for Unified World Modeling, Language Reasoning, and Action Synthesis
`WEM`	World-Ego Modeling for Long-Horizon Evolution in Hybrid Embodied Tasks
`HERMES++`	HERMES++: Toward a Unified Driving World Model for 3D Scene Understanding and Generation
`AstraNav-World`	AstraNav-World: World Model for Foresight Control and Consistency
`RynnVLA-002`	RynnVLA-002: A Unified Vision-Language-Action and World Model
`SWM`	Semantic World Models
`UniWM`	Unified World Models: Memory-Augmented Planning and Foresight for Visual Navigation
`F1`	F1: A Vision-Language-Action Model Bridging Understanding and Generation to Actions
`OccVLA`	OccVLA: Vision-Language-Action Model with Implicit 3D Occupancy Supervision
`VLWM`	Planning with Reasoning using Vision Language World Model
`DreamVLA`	DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge
`World4Omni`	World4Omni: A Zero-Shot Framework from Image Generation World Model to Robotic Manipulation
`WALL-E 2.0`	WALL-E 2.0: World Alignment by NeuroSymbolic Learning Improves World Model-based LLM Agent
`GR00T N1`	GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
`HERMES`	HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation
`Doe-1`	Doe-1: Closed-Loop Autonomous Driving with Large World Model
`Owl-1`	Owl-1: Omni World Model for Consistent Long Video Generation
`Eva`	Eva: An Embodied World Model for Future Video Anticipation
`PIVOT-R`	PIVOT-R: Primitive-Driven Waypoint-Aware World Model for Robotic Manipulation
`OccLLaMA`	OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Driving
`WorldGPT`	WorldGPT: Empowering LLM as Multimodal World Model
`3D-VLA`	3D-VLA: A 3D Vision-Language-Action Generative World Model
`ADriver-I`	ADriver-I: A General World Model for Autonomous Driving

1.2. Diffusion-based Generation

Latent Diffusion

Design	Paper	Link

`RynnWorld-4D`	RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation
`Holo-World`	Holo-World: Unified Camera, Object and Weather Control for Video World Model
`PAIWorld`	PAIWorld: A 3D-Consistent World Foundation Model for Robotic Manipulation
`Qwen-RobotWorld`	Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generation
`Kairos`	Kairos: A Regret-Aware Native World-Action Model Stack for Physical AI
`GEM-4D`	GEM-4D: Geometry-Enhanced Video World Models for Robot Manipulation
`RoboFlow4D`	RoboFlow4D: A Lightweight Flow World Model Toward Real-Time Flow-Guided Robotic Manipulation
`ReactiveGWM`	ReactiveGWM: Steering NPC in Reactive Game World Models
`Kinema4D`	Kinema4D: Kinematic 4D World Modeling for Spatiotemporal Embodied Simulation
`HyDRA`	Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Models
`PhyGenesis`	Toward Physically Consistent Driving Video World Models under Challenging Trajectories
`ABot-PhysWorld`	ABot-PhysWorld: Interactive World Foundation Model for Robotic Manipulation with Physics Alignment
`EVA`	EVA: Aligning Video World Models with Executable Robot Actions via Inverse Dynamics Rewards
`InSpatio-WorldFM`	InSpatio-WorldFM: An Open-Source Real-Time Generative Frame Model
`PlayWorld`	PlayWorld: Learning Robot World Models from Autonomous Play
`WorldCache`	WorldCache: Accelerating World Models for Free via Heterogeneous Token
`DreamWorld`	DreamWorld: Unified World Modeling in Video Generation
`Olaf-World`	Olaf-World: Orienting Latent Actions for Video World Modeling
`EgoWM`	Walk Through Paintings : Ego-centric World Models from Internet Priors
`ReWorld`	ReWorld: Multi-Dimensional Reward Modeling for Embodied World Models
`NeoVerse`	NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos
`GrndCtrl`	GrndCtrl: Grounding World Models via Self-Supervised Reward Alignment
`C^3`	World Models That Know When They Don’t Know: Controllable Video Generation with Calibrated Uncertainty
`GAIA-3`	GAIA-3: Scaling World Models to Power Safety and Evaluation
`WristWorld`	WristWorld: Generating Wrist-Views via 4D World Models for Robotic Manipulation
`WorldSplat`	WorldSplat: Gaussian-Centric Feed-Forward 4D Scene Generation for Autonomous Driving
`WorldForge`	WorldForge: Unlocking Emergent 3D/4D Generation in Video Diffusion Model via Training-Free Guidance
`PEWM`	Learning Primitive Embodied World Models: Towards Scalable Robotic Learning
`Video Policy`	Video Generators are Robot Policies
`Ego-PM`	Ego-centric Predictive Model Conditioned on Hand Trajectories
`GWM`	GWM: Towards Scalable Gaussian World Models for Robotic Manipulation
`M3arsSynth`	Martian World Models: Controllable Video Synthesis with Physically Accurate 3D Reconstructions
`AirScape`	AirScape: An Aerial Generative World Model with Motion Controllability
`robot4dgen`	Geometry-aware 4D Video Generation for Robot Manipulation
`Genesis`	Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistency
`MinD`	MinD: Learning A Dual-System World Model for Real-Time Planning and Implicit Risk Analysis
`RealPlay`	PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling
`COME`	COME: Adding Scene-Centric Forecasting Control to Occupancy World Model
`MeWM`	Medical World Model: Generative Simulation of Tumor Evolution for Treatment Planning
`StateSpaceDiffuser`	StateSpaceDiffuser: Bringing Long Context to Diffusion World Models
`GeoDrive`	GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Control
`3DPEWM`	Learning 3d persistent embodied world models
`RoboTransfer`	RoboTransfer: Geometry-Consistent Video Diffusion for Robotic Visual Policy Transfer
`FlowDreamer`	FlowDreamer: A RGB-D World Model with Flow-based Motion Representations for Robot Manipulation
`LaDi-WM`	LaDi-WM: A Latent Diffusion-based World Model for Predictive Manipulation
`LangToMo`	Pixel motion as universal representation for robot control
`DreamGen`	DreamGen: Unlocking Generalization in Robot Learning through Video World Models
`Learning to Drive`	Learning to Drive from a World Model
`Tesseract`	Tesseract: Learning 4d embodied world models
`UWM`	Unified world models: Coupling video and action diffusion for pretraining on large robotic datasets
`ViMo`	ViMo: A Generative Visual GUI World Model for App Agents
`DiST-4D`	DiST-4D: Disentangled Spatiotemporal Diffusion with Metric Depth for 4D Driving Scene Generation
`Aether`	Aether: Geometric-aware unified world modeling
`Cosmos transfer1`	Cosmos transfer1: Conditional world generation with adaptive multimodal control
`GAIA-2`	GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving
`EDELINE`	EDELINE: Enhancing Memory in Diffusion-based World Models via Linear-Time Sequence Modeling
`MaskGWM`	MaskGWM: A Generalizable Driving World Model with Video Mask Reconstruction
`Dreamdrive`	Dreamdrive: Generative 4d scene modeling from street view images
`Cosmos`	Cosmos World Foundation Model Platform for Physical AI
`VPP`	Video prediction policy: A generalist robot policy with predictive visual representations
`Imagine-2-drive`	Imagine-2-drive: High-fidelity world modeling in carla for autonomous vehicles
`GenEx`	Generative World Explorer
`DOME`	DOME: Taming Diffusion Model into High-Fidelity Controllable Occupancy World Model
`Panacea+`	Panacea+: Panoramic and Controllable Video Generation for Autonomous Driving
`Bevworld`	Bevworld: A multimodal world model for autonomous driving via unified bev latent space
`Delphi`	Unleashing generalization of end-to-end autonomous driving with controllable long video generation
`Occsora`	Occsora: 4d occupancy generation models as world simulators for autonomous driving
`GenAD`	Generalized Predictive Model for Autonomous Driving
`WorldGPT`	Worldgpt: a sora-inspired video ai agent as rich world models from text and image inputs
`DriveDreamer-2`	DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation
`WoVoGen`	WoVoGen: World Volume-Aware Diffusion for Controllable Multi-Camera Driving Scene Generation
`Drive-WM`	Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving
`Panacea`	Panacea: Panoramic and controllable video generation for autonomous driving
`Drivingdiffusion`	DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion model
`DriveDreamer`	DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving
`UniPi`	Learning universal policies via text-guided video generation

Autoregressive Diffusion

Design	Paper	Link

`MemLearner`	MemLearner: Learning to Query Context memory for Video World Models
`ActWorld`	ActWorld: From Explorable to Interactive World Model via Action-Aware Memory
`DreamX-World`	DreamX-World 1.0: A General-Purpose Interactive World Model
`Mirage`	Latent Spatial Memory for Video World Models
`Echo-Memory`	Echo-Memory: A Controlled Study of Memory in Action World Models
`Cosmos 3`	Cosmos 3: Omnimodal World Models for Physical AI
`Gamma-World`	Gamma-World: Generative Multi-Agent World Modeling Beyond Two Players
`SANA-WM`	SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformer
`Matrix-Game 3.0`	Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory
`WorldCam`	WorldCam: Interactive Autoregressive 3D Gaming Worlds with Camera Pose as a Unifying Geometric Representation
`SWM`	Grounding World Simulation Models in a Real-World Metropolis
`LIVE`	LIVE: Long-horizon Interactive Video World Modeling
`LingBot-World`	Advancing Open-source World Models
`UniDrive-WM`	UniDrive-WM: Unified Understanding, Planning and Generation World Model For Autonomous Driving
`Yume1.5`	Yume1.5: A Text-Controlled Interactive World Generation Model
`HY-World 1.5`	HY-World 1.5: A Systematic Framework for Interactive World Modeling with Real-Time Latency and Geometric Consistency
`Astra`	Astra: General Interactive World Model with Autoregressive Denoising
`RELIC`	RELIC: Interactive Video World Model with Long-Horizon Memory
`ANWM`	Aerial World Model for Long-horizon Visual Generation and Navigation in 3D Space
`WorldPack`	WorldPack: Compressed Memory Improves Spatial Consistency in Video World Modeling
`Hunyuan-GameCraft-2`	Hunyuan-GameCraft-2: Instruction-following Interactive Game World Model
`PAN`	PAN: A World Model for General, Interactable, and Long-Horizon World Simulation
`Emu3.5`	Emu3.5: Native Multimodal Models are World Learners
`OmniNWM`	OmniNWM: Omniscient Driving Navigation World Models
`WoW`	WoW: Towards a World omniscient World model Through Embodied Interaction
`LongScape`	PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling
`Dreamer V4`	Training Agents Inside of Scalable World Models
`Genie Envisioner`	Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulation
`LiDARCrafter`	LiDARCrafter: Dynamic 4D World Modeling from LiDAR Sequences
`Yan`	Yan: Foundational Interactive Video Generation
`Matrix-Game 2.0`	Matrix-Game 2.0: An Open-Source, Real-Time, and Streaming Interactive World Model
`Yume`	Yume: An Interactive World Generation Model
`Geometry Forcing`	Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modeling
`spmem`	Video World Models with Long-term Spatial Memory
`STAGE`	STAGE: A Stream-Centric Generative World Model for Long-Horizon Driving-Scene Simulation
`CaM`	Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrieval
`DeepVerse`	DeepVerse: 4D Autoregressive Video Generation as a World Model
`Epona`	Epona: Autoregressive Diffusion World Model for Autonomous Driving
`SceneDiffuser++`	SceneDiffuser++: City-Scale Traffic Simulation via a Generative World Model
`VMem`	VMem: Consistent Interactive Video Scene Generation with Surfel-Indexed View Memory
`Hunyuan-GameCraft`	Hunyuan-GameCraft: High-dynamic Interactive Game Video Generation with Hybrid History Condition
`Matrix-Game`	Matrix-Game: Interactive World Foundation Model
`PEVA`	Whole-Body Conditioned Egocentric Video Prediction
`NFD`	Playing with Transformer at 30+ FPS via Next-Frame Diffusion
`VRAG`	Learning World Models for Interactive Video Generation
`DriVerse`	DriVerse: Navigation World Model for Driving Simulation via Multimodal Trajectory Prompting and Motion Alignment
`WORLDMEM`	WORLDMEM: Long-term Consistent World Simulation with Memory
`Adaworld`	Adaworld: Learning adaptable world models with latent actions
`TSWM`	Toward Stable World Models: Measuring and Addressing World Instability in Generative Environments
`Gamefactory`	Gamefactory: Creating new games with generative interactive videos
`PlayGen`	Playable Game Generation
`InfinityDrive`	InfinityDrive: Breaking Time Limits in Driving World Models
`GEM`	Gem: A generalizable ego-vision multimodal world model for fine-grained ego-motion, object dynamics, and scene composition control
`UniMLVG`	UniMLVG: Unified Framework for Multi-view Long Video Generation with Comprehensive Control Capabilities for Autonomous Driving
`The Matrix`	The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control
`NWM`	Navigation World Models
`Copilot4D`	Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion
`Drivingsphere`	Drivingsphere: Building a high-fidelity 4d world for closed-loop simulation
`Gamegen-x`	Gamegen-x: Interactive open-world game video generation
`Genie 2`	Genie 2: A large-scale foundation world model
`oasis`	Oasis: A Universe in a Transformer
`GameNGen`	Diffusion models are real-time game engines
`IRAsim`	IRASim: A Fine-Grained World Model for Robot Manipulation
`diamond`	Diffusion for World Modeling: Visual Details Matter in Atari
`Vista`	Vista: A generalizable driving world model with high fidelity and versatile controllability
`UniSim`	UniSim: Learning Interactive Real-World Simulators

1.3. Embedding Prediction

JEPA

Design	Paper	Link
`AdaJEPA`	AdaJEPA: An Adaptive Latent World Model
`FR3D`	Future Dynamic 3D Reconstruction: A 3D World Model with Disentangled Ego-Motion
`Sub-JEPA`	Sub-JEPA: Subspace Gaussian Regularization for Stable End-to-End World Models
`HWM`	Hierarchical Planning with Latent World Models
`LeWorldModel`	LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels
`V-JEPA 2.1`	V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning
`Temporal Straightening`	Temporal Straightening for Latent Planning
`C-JEPA`	Causal-JEPA: Learning World Models through Object-Level Latent Interventions
`VLA-JEPA`	VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Model
`DDP-WM`	DDP-WM: Disentangled Dynamics Prediction for Efficient World Models
`DINO-world`	Back to the Features: DINO as a Foundation for Video World Models
`V-JEPA 2`	V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
`SIVWM`	Sparse Imagination for Efficient Visual World Model Planning
`seq-JEPA`	seq-JEPA: Autoregressive Predictive Learning of Invariant-Equivariant World Models
`Flare`	Flare: Robot learning with implicit world modeling
`OSVI-WM`	OSVI-WM: One-Shot Visual Imitation for Unseen Tasks using World-Model-Guided Trajectory Generation
`EchoWorld`	EchoWorld: Learning Motion-Aware World Models for Echocardiography Probe Guidance
`AD-L-JEPA`	AD-L-JEPA: Self-Supervised Spatial World Models with Joint Embedding Predictive Architecture for Autonomous Driving with LiDAR Data
`DINO-Foresight`	DINO-Foresight: Looking into the Future with DINO
`DINO-WM`	DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
`LAW`	Enhancing End-to-End Autonomous Driving with Latent World Model
`V-JEPA`	Revisiting Feature Prediction for Learning Visual Representations from Video
`IWM`	Learning and Leveraging World Models in Visual Representation Learning

1.4. State Transition

Latent State-Space Modeling

Design	Paper	Link

`NE-Dreamer`	Next Embedding Prediction Makes World Models Stronger
`BOOM`	Bootstrap Off-policy with World Model
`GASv2`	Visuomotor Grasping with World Models for Surgical Robots
`LPS`	Latent Policy Steering with Embodiment-Agnostic Pretrained World Models
`EMERALD`	Accurate and Efficient World Modeling with Masked Latent Transformers
`FOUNDER`	FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Making
`NavMorph`	NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environments
`Raw2Drive`	Raw2Drive: Reinforcement Learning with Aligned World Models for End-to-End Autonomous Driving
`SSVWM`	Long-Context State-Space Video World Models
`PIN-WM`	PIN-WM: Learning Physics-Informed World Models for Non-Prehensile Manipulation
`ReDRAW`	Adapting World Models with Latent-State Dynamics Residuals
`WoTE`	End-to-End Driving with Online Trajectory Evaluation via BEV World Model
`DyWA`	DyWA: Dynamics-Adaptive World Action Model for Generalizable Non-Prehensile Manipulation
`DMWM`	DMWM: Dual-Mind World Model with Long-Term Imagination
`Simulus`	Uncovering Untapped Potential in Sample-Efficient World Model Agents
`S5WM`	Accelerating Model-Based Reinforcement Learning with State-Space World Models
`AdaWM`	AdaWM: Adaptive World Model based Planning for Autonomous Driving
`RoboHorizon`	RoboHorizon: An LLM-Assisted Multi-View World Model for Long-Horizon Robotic Manipulation
`LS-Imagine`	Open-World Reinforcement Learning over Long Short-Term Imagination
`-`	Mitigating Covariate Shift in Imitation Learning for Autonomous Vehicles Using Latent Space Generative World Models
`GenRL`	GenRL: Multimodal-Foundation World Models for Generalization in Embodied Agents
`DriveWorld`	DriveWorld: 4D Pre-Trained Scene Understanding via World Models for Autonomous Driving
`Puppeteer`	Hierarchical World Models as Visual Whole-Body Humanoid Controllers
`R2I`	Mastering Memory Tasks with World Models
`REM`	Improving Token-Based World Models with Parallel Observation Prediction
`Think2Drive`	Think2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving
`MUVO`	MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representations
`STORM`	STORM: Efficient Stochastic Transformer based World Models for Reinforcement Learning
`HarmonyDream`	HarmonyDream: Task Harmonization Inside World Models
`TD-MPC2`	TD-MPC2: Scalable, Robust World Models for Continuous Control
`MoDem-V2`	MoDem-V2: Visuo-Motor World Models for Real-World Robot Manipulation
`SWIM`	Structured World Models from Human Videos
`Dynalang`	Learning to Model the World with Language
`SafeDreamer`	SafeDreamer: Safe Reinforcement Learning with World Models
`CoWorld`	Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learning
`TWM`	Transformer-Based World Models Are Happy With 100k Interactions
`MV-MWM`	Multi-View Masked World Models for Visual Robotic Manipulation
`DreamV3`	Mastering Diverse Domains through World Models
`IRIS`	Transformers are Sample-Efficient World Models
`MWM`	Masked World Models for Visual Control
`DayDreamer`	DayDreamer: World Models for Physical Robot Learning
`CADDY`	Playable Video Generation
`DreamV2`	Mastering Atari with Discrete World Models
`DreamV1`	Dream to Control: Learning Behaviors by Latent Imagination
`PlaNet`	Learning Latent Dynamics for Planning from Pixels

Object-Centric Modeling

Design	Paper	Link

`LPWM`	Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modeling
`FIOC-WM`	Learning Interactive World Model for Object-Centric Reinforcement Learning
`Dyn-o`	Dyn-O: Building Structured World Models with Object-Centric Representations
`SlotPi`	SlotPi: Physics-informed Object-centric Reasoning Models
`-`	Object-Centric World Model for Language-Guided Manipulation
`DisWM`	Disentangled World Models: Learning to Transfer Semantic Knowledge from Distracting Videos for Reinforcement Learning
`Objects matter`	Objects matter: object-centric world models improve reinforcement learning in visually complex environments
`Dreamweaver`	Dreamweaver: Learning Compositional World Models from Pixels
`MEAD`	Efficient Exploration and Discriminative World Model Learning with an Object-Centric Abstraction
`slotSSM`	Slot State Space Models
`RoboDreamer`	RoboDreamer: Learning Compositional World Models for Robot Imagination
`SSWM`	Slot Structured World Models
`cosmos`	Neurosymbolic Grounding for Compositional World Models
`FOCUS`	FOCUS: Object-Centric World Models for Robotics Manipulation
`SlotFormer`	SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models
`HOWM`	Toward Compositional Generalization in Object-Oriented World Modeling
`G-SWM`	Improving Generative Imagination in Object-Centric World Models

1.5. Other Architectures

Design	Paper	Link

`PhysWorld`	PhysWorld: From Real Videos to World Models of Deformable Objects via Physics-Aware Demonstration Synthesis
`FASTopoWM`	FASTopoWM: Fast-Slow Lane Segment Topology Reasoning with Latent World Models
`GWM`	Graph World Model
`World4Drive`	World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Model
`Orbis`	Orbis: Overcoming Challenges of Long-Horizon Prediction in Driving World Models
`LiDARWM`	Towards foundational LiDAR world models with efficient latent flow matching
`ManiGaussian++`	ManiGaussian++: General Robotic Bimanual Manipulation with Hierarchical Gaussian World Model
`GAF`	GAF: Gaussian Action Field as a 4D Representation for Dynamic World Modeling in Robotic Manipulation
`HWM`	Humanoid World Models: Open World Foundation Models for Humanoid Robotics
`DriveX`	DriveX: Omni Scene Modeling for Learning Generalizable World Knowledge in Autonomous Driving
`OccProphet`	OccProphet: Pushing Efficiency Frontier of Camera-Only 4D Occupancy Forecasting with Observer-Forecaster-Refiner Framework
`FleetWM`	Multi-Task Interactive Robot Fleet Learning with Visual World Models
`GaussianWorld`	GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Prediction
`NeMo`	Neural volumetric world models for autonomous driving

2. Datasets & Benchmarks

2.1. Foundational World Modeling

General World Prediction and Simulation

Design	Paper	Link

`MemoBench`	MemoBench: Benchmarking World Modeling in Dynamically Changing Environments
`WRBench`	Current World Models Lack a Persistent State Core
`MBench`	MBench: A Comprehensive Benchmark on Memory Capability for Video World Models
`WBench`	WBench: A Comprehensive Multi-turn Benchmark for Interactive Video World Model Evaluation
`iWorld-Bench`	iWorld-Bench: A Benchmark for Interactive World Models with a Unified Action Generation Framework
`WR-Arena`	World Reasoning Arena
`CoW-Bench`	The Trinity of Consistency as a Defining Principle for General World Models
`MIND`	MIND: Benchmarking Memory Consistency and Action Control in World Models
`DynamicVerse`	DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modeling
`4DWorldBench`	4DWorldBench: A Comprehensive Evaluation Framework for 3D/4D World Generation Models
`Gen-ViRe`	Can World Simulators Reason? Gen-ViRe: A Generative Visual Reasoning Benchmark
`World-in-World`	World-in-World: World Models in a Closed-Loop World
`OmniWorld`	OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modeling
`Sekai`	Sekai: A Video Dataset towards World Exploration
`WorldPrediction`	WorldPrediction: A Benchmark for High-level World Modeling and Long-horizon Procedural Planning
`WorldScore`	WorldScore: A Unified Evaluation Benchmark for World Generation
`MM-OR`	MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environments
`WorldModelBench`	WorldModelBench: Judging Video Generation Models As World Models
`OpenHumanVid`	OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generation
`EgoVid-5M`	EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generation
`Ego-Exo4D`	Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
`InternVid`	InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation
`Ego4D`	Ego4D: Around the World in 3,000 Hours of Egocentric Video
`WebVid-2M`	Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval
`EPIC-KITCHENS-100`	Rescaling Egocentric Vision
`HowTo100M`	HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
`COIN`	COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis
`SSV2`	The "something something" video database for learning and evaluating visual common sense
`Kinetics`	The Kinetics Human Action Video Dataset
`YouTube-8M`	YouTube-8M: A Large-Scale Video Classification Benchmark
`UCF101`	UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild

Physics and Causality Benchmark

Design	Paper	Link

`PhysEditWorld`	PhysEditWorld: A Large-Scale Dataset Toward Physics-Editable World Models
`Omni-WorldBench`	Omni-WorldBench: Towards a Comprehensive Interaction-Centric Evaluation for World Models
`WorldBench`	WorldBench: Disambiguating Physics for Diagnostic Evaluation of World Models
`VideoVerse`	VideoVerse: How Far is Your T2V Generator from a World Model?
`PAI-Bench`	Physical ai bench: A comprehensive benchmark for physical ai generation and understanding
`PhysVidBench`	Can Your Model Separate Yolks with a Water Bottle? Benchmarking Physical Commonsense Understanding in Video Generation Models
`PBench`	PBench: A benchmark for evaluating generative models
`IntPhys 2`	IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environments
`T2VPhysBench`	T2VPhysBench: A First-Principles Benchmark for Physical Consistency in Text-to-Video Generation
`PisaBench`	PISA Experiments: Exploring Physics Post-Training for Video Diffusion Models by Watching Stuff Drop
`WISA-32K`	WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generation
`VideoPhy-2`	VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation
`VBench-2.0`	VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness
`PhyCoBench`	A Physical Coherence Benchmark for Evaluating Video Generation Models via Optical Flow-guided Frame Prediction
`Physics-IQ`	Do generative video models understand physical principles?
`PhyGenBench`	Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation
`PhyBench`	PhyBench: A Physical Commonsense Benchmark for Evaluating Text-to-Image Models
`VideoPhy`	VideoPhy: Evaluating Physical Commonsense for Video Generation
`Physion++`	Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Properties
`InfLevel`	Benchmarking Progress to Infant-Level Physical Reasoning in AI
`VoE`	A Benchmark for Modeling Violation-of-Expectation in Physical Reasoning Across Event Categories
`Physion`	Physion: Evaluating Physical Prediction from Vision in Humans and Machines
`CoPhy`	CoPhy: Counterfactual Learning of Physical Dynamics
`IntPhys`	IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning

2.2. Domain-specific World Modeling

Embodied AI and Robotics

Design	Paper	Link

`WMBench (GigaWorld-1)`	GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation
`WorldArena 2.0`	WorldArena 2.0: Extending Embodied World Model Benchmarking on Modality, Functionality and Platform
`GE-Sim 2.0`	Genie Envisioner World Simulator 2.0
`RoboWM-Bench`	RoboWM-Bench: A Benchmark for Evaluating World Models in Robotic Manipulation
`WorldArena`	WorldArena: A Unified Benchmark for Evaluating Perception and Functional Utility of Embodied World Models
`DreamDojo`	DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos
`WoW-World-Eval`	Wow, wo, val! A Comprehensive Embodied World Model Evaluation Turing Test
`GigaWorld-0`	GigaWorld-0: World Models as Data Engine to Empower Embodied AI
`Target-Bench`	Target-Bench: Can World Models Achieve Mapless Path Planning with Semantic Targets
`WoW`	WoW: Towards a world omniscient world model through embodied interaction
`Meta-World+`	Meta-World+: An improved, standardized, rl benchmark
`AgiBot-World`	AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems
`RoboCasa`	RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots
`DROID`	DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
`BEHAVIOR-1K`	BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation
`MimicGen`	MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations
`OXE`	Open X-Embodiment: Robotic Learning Datasets and RT-X Models
`RH20T`	RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot
`VP²`	A Control-Centric Benchmark for Video Prediction
`RT-1`	RT-1: Robotics Transformer for Real-World Control at Scale
`BC-Z`	BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning
`CALVIN`	CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks
`BridgeData V2`	BridgeData V2: A Dataset for Robot Learning at Scale
`LIBERO`	LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning
`Isaac Gym`	Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning
`RoboNet`	RoboNet: Large-Scale Multi-Robot Learning
`Meta-World`	Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
`RLBench`	RLBench: The Robot Learning Benchmark & Learning Environment

Autonomous Driving

Design	Paper	Link

`WorldLens`	WorldLens: Full-Spectrum Evaluations of Driving World Models in Real World
`ACT-Bench`	ACT-Bench: Towards Action Controllable World Models for Autonomous Driving
`DrivingDojo`	DrivingDojo Dataset: Advancing Interactive and Knowledge-Enriched Driving World Model
`DriveArena`	DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Driving
`NAVSIM`	NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking
`CarDreamer`	CarDreamer: Open-Source Learning Platform for World Model based Autonomous Driving
`OpenDV-2K`	GenAD: Generalized Predictive Model for Autonomous Driving
`ZOD`	Zenseact Open Dataset: A Large-Scale and Diverse Multimodal Dataset for Autonomous Driving
`Occ3D`	Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving
`OpenOccupancy`	OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perception
`Argoverse 2`	Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting
`KITTI-360`	KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D
`NuPlan`	NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles
`ONCE`	One Million Scenes for Autonomous Driving: ONCE Dataset
`WOMD`	Large Scale Interactive Motion Forecasting for Autonomous Driving: The Waymo Open Motion Dataset
`Lyft Level 5`	One Thousand and One Hours: Self-driving Motion Prediction Dataset
`A2D2`	A2D2: Audi Autonomous Driving Dataset
`Waymo`	Scalability in Perception for Autonomous Driving: Waymo Open Dataset
`Argoverse`	Argoverse: 3D Tracking and Forecasting With Rich Maps
`INTERACTION`	INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps
`SemanticKITTI`	SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences
`nuScenes`	nuScenes: A Multimodal Dataset for Autonomous Driving
`BDD100K`	BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning
`ApolloScape`	The ApolloScape Dataset for Autonomous Driving
`CARLA`	CARLA: An Open Urban Driving Simulator
`KITTI`	Are we ready for autonomous driving? The KITTI vision benchmark suite

Interactive Environments and Gaming

Design	Paper	Link

`SCOPE`	SCOPE: Simulating Cross-game Operations in Playable Environments for FPS World Models
`WorldMark`	WorldMark: A Unified Benchmark Suite for Interactive Video World Models
`WildWorld`	WildWorld: A Large-Scale Dataset for Dynamic World Modeling with Actions and Explicit State toward Generative ARPG
`Matrix-Game-MC`	Matrix-Game: Interactive World Foundation Model
`LOOPNAV`	Toward Memory-Aided World Models: Benchmarking via Spatial Consistency
`JARVIS-VLA`	JARVIS-VLA: Post-Training Large-Scale Visual Language Models to Play Visual Games with Keyboards and Mouse
`GF-Minecraft`	GameFactory: Creating New Games with Generative Interactive Videos
`The Matrix`	The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control
`GameGen-X`	GameGen-X: Interactive Open-world Game Video Generation
`Mars`	Mars: Situated Inductive Reasoning in an Open-World Environment
`Crafter`	Benchmarking the Spectrum of Agent Capabilities
`CS-Deathmatch`	Counter-Strike Deathmatch with Large-Scale Behavioural Cloning
`MineRL`	The MineRL 2020 Competition on Sample Efficient Reinforcement Learning using Human Priors
`NLE`	The NetHack Learning Environment
`Procgen`	Leveraging Procedural Generation to Benchmark Reinforcement Learning
`DMC`	DeepMind Control Suite
`ALE`	The Arcade Learning Environment: An Evaluation Platform for General Agents

3. Others

Survey

Paper	Link

World Model for Robot Learning: A Comprehensive Survey
OpenWorldLib: A Unified Codebase and Definition of Advanced World Models
Video Generation Models as World Models: Efficient Paradigms, Architectures and Algorithms
Learning to Model the World: A Survey of World Models in Artificial Intelligence
Towards Generalist Embodied AI: A Survey on World Models for VLA Agents
Video Generation Models in Robotics -- Applications, Research Challenges, Future Directions
Digital Twin AI: Opportunities and Challenges from Large Language Models to World Models
Progressive Robustness-Aware World Models in Autonomous Driving: A Review and Outlook
Simulating the Visual World with Artificial Intelligence: A Roadmap
A Step Toward World Models: A Survey on Robotic Manipulation
The Safety Challenge of World Models for Embodied AI Agents: A Review
A Comprehensive Survey on World Models for Embodied AI
3D and 4D World Modeling: A Survey
Large Model Empowered Embodied AI: A Survey on Decision-Making and Embodied Learning
A Survey: Learning Embodied Intelligence from Physical Simulators and World Models
From 2D to 3D Cognition: A Brief Survey of General World Models
A Survey on World Models Grounded in Acoustic Physical Information
World Models in Artificial Intelligence: Sensing, Learning, and Reasoning Like a Child
The Role of World Models in Shaping Autonomous Driving: A Comprehensive Survey
A Survey of World Models for Autonomous Driving
Understanding World or Predicting Future? A Comprehensive Survey of World Models
Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyond
World Models for Autonomous Driving: An Initial Survey
A Path Towards Autonomous Machine Intelligence

GitHub Repo

This part overlaps somewhat with the Survey section.

Repo	Link

WMFactory 0.5: One environment · One procedure · Eleven interactive world models
A minimalist repository for training video world models based on diffusion-forcing
Awesome Video World Models with AR Diffusion
Awesome From Video Generation to World Model
A Curated List of Amazing Works in World Modeling, spanning applications in Embodied AI, Autonomous Driving, Natural Laguage Processing and Agents
A curated list of papers for World Models for General Video Generation, Embodied AI, and Autonomous Driving
World Models for Autonomous Driving (and Robotic) papers
Awesome 3D and 4D World Models
Simulating the Real World: Survey & Resources
A curated list of world models for autonomous driving
A curated list of resources related to World Models for Autonomous Driving (WMAD)
A Survey: Learning Embodied Intelligence from Physical Simulators and World Models
A Comprehensive Survey on World Models for Embodied AI
A Survey on World Models Grounded in Acoustic Physical Information
A Comprehensive Survey on General World Models and Beyond
Understanding World or Predicting Future? A Comprehensive Survey of World Models
From Masks to Worlds: A Hitchhiker's Guide to World Models

Workshop

Venue	Workshop	Link

`ECCV 2026`	How to Build Effective World Models for Embodied AI
`CVPR 2026`	GigaBrain Challenge 2026: World Models & VLA for Embodied Intelligence
`CVPR 2026`	Video World Models
`CVPR 2026`	World Models Meet Active Sensing and Closed-Loop Planning
`ICLR 2026`	Workshop on World Models: Understanding, Modelling and Scaling
`NeurIPS 2025`	Workshop on Bridging Language, Agent, and World Models for Reasoning and Planning
`NeurIPS 2025`	Embodied World Models for Decision Making
`ICCV 2025`	Reliable and Interactable World Models: Geometry, Physics, Interactivity and Real-World Generalization
`ICML 2025`	Building Physically Plausible World Models
`CVPR 2025`	Benchmarking World Models
`ICLR 2025`	World Models: Understanding, Modelling, and Scaling

Theory

Design	Paper	Link

`Orca`	Orca: The World is in Your Mind
`LoopWM`	Looped World Models
`OrbiSim`	OrbiSim: World Models as Differentiable Physics Engines for Embodied Intelligence
`-`	Physically Native World Models: A Hamiltonian Perspective on Generative World Modeling
`ZWM`	Zero-shot World Models Are Developmentally Efficient Learners
`-`	Human Cognition in Machines: A Unified Perspective of World Models
`EB-JEPA`	A Lightweight Library for Energy-Based Joint-Embedding Predictive Architectures
`-`	Research on World Models Is Not Merely Injecting World Knowledge into Specific Tasks
`-`	What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?
`-`	Closing the Train-Test Gap in World Models for Gradient-Based Planning
`WWM`	Web World Model
`DWM`	Dexterous World Models	[](https://github.com/s

Truncated — view the full README on GitHub.

Contributors

33 commits

2 commits

2 commits

1 commits

Similar repos

JiahuaDong/Awesome-World-Models

353

ziqihuangg/Awesome-From-Video-Generation-to-World-Model

A list of works on video generation towards world model

518

liujiuming123/Awesome-Interactive-World-Model

A Comprehensive Survey of Interactive Video World Models

234

knightnemo/Awesome-World-Models

A Curated List of Awesome Works in World Modeling, Aiming to Serve as a One-stop Resource for…

3,441

NTUMARS/Awesome-World-Model-for-Robotics-Policy

811

ALEEEHU/World-Simulator

[IEEE TPAMI 2026] Simulating the Real World: Survey & Resources, which contains our survey…

389

yueen-ma/Awesome-VLA

671

HaoranZhuExplorer/World-Models-Autonomous-Driving-Survey

A curated list of world models for autonomous driving.

535

AIWorldLab/Awesome-Vision-World-Model

From Seeing to Knowing the World: A Survey of Vision World Models

193

38 commits

updated Aug 18, 2026

AIWorldLab.github.io/survey

See the code

README

Awesome Vision World Models

This repository accompanies our survey From Seeing to Knowing the World: A Survey of Vision World Models and maintains a structured collection of Vision World Model resources:

It organizes papers across 7 Vision World Model designs and 5 dataset/benchmark categories, with their corresponding arXiv IDs, GitHub repositories, and project pages.
It also collects broader resources for world modeling, including theoretical analyses, top-tier conference workshops, interesting repositories, downstream-task applications, and other useful perspectives.

For more details, kindly refer to our paper :rocket:

:newspaper: News

2026-07-07: Add 27 papers, including 19 world model methods and 8 datasets/benchmarks. Update the papers in the teaser figure through 2026-07-01.
2026-05-23: Add 16 papers, 3 github repositories and 5 workshops.

More updates

2026-05-18: Fix metadata for 2 papers.
2026-04-08: Add 45 papers.

1. Vision World Model Designs
2. Datasets & Benchmarks
- 2.1. Foundational World Modeling
  - General World Prediction and Simulation
  - Physics and Causality Benchmark
- 2.2. Domain-specific World Modeling
3. Others
4. Acknowledgements
Citations

1. Designs

1.1. Sequential Generation

Visual Autoregressive Modeling

Design	Paper	Link

`VideoWorld 2`	VideoWorld 2: Learning Transferable Knowledge from Real-world Videos
`iMoWM`	iMoWM: Taming Interactive Multi-Modal World Model for Robotic Manipulation
`PWM`	From Forecasting to Planning: Policy World Model for Collaborative State-Action Prediction
`SAMPO`	SAMPO: Scale-wise Autoregression with Motion Prompt for Generative World Models
`RynnVLA-001`	RynnVLA-001: Using Human Demonstrations to Improve Robot Manipulation
`OccTENS`	OccTENS: 3D Occupancy World Model via Temporal Next-Scale Prediction
`Genie 3`	Genie 3: A new frontier for world models
`I²-world`	I²-World: Intra-Inter Tokenization for Efficient Dynamic 4D Scene Forecasting
`UniVLA`	Unified Vision-Language-Action Model
`WorldVLA`	WorldVLA: Towards Autoregressive Action World Model
`RoboScape`	RoboScape: Physics-informed Embodied World Model
`Xray2Xray`	Xray2Xray: World Model from Chest X-rays with Volumetric Context
`RLVR-World`	RLVR-World: Training World Models with Reinforcement Learning
`MineWorld`	MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft
`UVA`	Unified Video Action Model
`SurgWM`	Surgical Vision World Model
`DWS`	Pre-Trained Video Generative Models as World Simulators
`VideoWorld`	VideoWorld: Exploring Knowledge Learning from Unlabeled Videos
`Drivingworld`	Drivingworld: Constructing world model for autonomous driving via video GPT
`Moto`	Moto: Latent motion token as the bridging language for learning robot manipulation from video
`WHALE`	WHALE: Towards Generalizable and Scalable World Models for Embodied Decision-making
`GR-2`	GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation
`LatentDriver`	Learning Multiple Probabilistic Decisions from Latent World Model in Autonomous Driving
`Renderworld`	Renderworld: World model with self-supervised 3D label
`VidIT`	Video In-context Learning: Autoregressive Transformers are Zero-Shot Video Imitators
`iVideoGPT`	iVideoGPT: Interactive VideoGPTs are Scalable World Models
`Genie`	Genie: Generative Interactive Environments
`LWM`	World Model on Million-Length Video And Language With Blockwise RingAttention
`WHAM`	World and human action models towards gameplay ideation
`WorldDreamer`	WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokens
`OccWorld`	OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving
`GAIA-1`	Gaia-1: A generative world model for autonomous driving

MLLM-guided Multimodal Autoregressive Model

Design	Paper	Link

`WLA`	World-Language-Action Model for Unified World Modeling, Language Reasoning, and Action Synthesis
`WEM`	World-Ego Modeling for Long-Horizon Evolution in Hybrid Embodied Tasks
`HERMES++`	HERMES++: Toward a Unified Driving World Model for 3D Scene Understanding and Generation
`AstraNav-World`	AstraNav-World: World Model for Foresight Control and Consistency
`RynnVLA-002`	RynnVLA-002: A Unified Vision-Language-Action and World Model
`SWM`	Semantic World Models
`UniWM`	Unified World Models: Memory-Augmented Planning and Foresight for Visual Navigation
`F1`	F1: A Vision-Language-Action Model Bridging Understanding and Generation to Actions
`OccVLA`	OccVLA: Vision-Language-Action Model with Implicit 3D Occupancy Supervision
`VLWM`	Planning with Reasoning using Vision Language World Model
`DreamVLA`	DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge
`World4Omni`	World4Omni: A Zero-Shot Framework from Image Generation World Model to Robotic Manipulation
`WALL-E 2.0`	WALL-E 2.0: World Alignment by NeuroSymbolic Learning Improves World Model-based LLM Agent
`GR00T N1`	GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
`HERMES`	HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation
`Doe-1`	Doe-1: Closed-Loop Autonomous Driving with Large World Model
`Owl-1`	Owl-1: Omni World Model for Consistent Long Video Generation
`Eva`	Eva: An Embodied World Model for Future Video Anticipation
`PIVOT-R`	PIVOT-R: Primitive-Driven Waypoint-Aware World Model for Robotic Manipulation
`OccLLaMA`	OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Driving
`WorldGPT`	WorldGPT: Empowering LLM as Multimodal World Model
`3D-VLA`	3D-VLA: A 3D Vision-Language-Action Generative World Model
`ADriver-I`	ADriver-I: A General World Model for Autonomous Driving

1.2. Diffusion-based Generation

Latent Diffusion

Design	Paper	Link

`RynnWorld-4D`	RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation
`Holo-World`	Holo-World: Unified Camera, Object and Weather Control for Video World Model
`PAIWorld`	PAIWorld: A 3D-Consistent World Foundation Model for Robotic Manipulation
`Qwen-RobotWorld`	Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generation
`Kairos`	Kairos: A Regret-Aware Native World-Action Model Stack for Physical AI
`GEM-4D`	GEM-4D: Geometry-Enhanced Video World Models for Robot Manipulation
`RoboFlow4D`	RoboFlow4D: A Lightweight Flow World Model Toward Real-Time Flow-Guided Robotic Manipulation
`ReactiveGWM`	ReactiveGWM: Steering NPC in Reactive Game World Models
`Kinema4D`	Kinema4D: Kinematic 4D World Modeling for Spatiotemporal Embodied Simulation
`HyDRA`	Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Models
`PhyGenesis`	Toward Physically Consistent Driving Video World Models under Challenging Trajectories
`ABot-PhysWorld`	ABot-PhysWorld: Interactive World Foundation Model for Robotic Manipulation with Physics Alignment
`EVA`	EVA: Aligning Video World Models with Executable Robot Actions via Inverse Dynamics Rewards
`InSpatio-WorldFM`	InSpatio-WorldFM: An Open-Source Real-Time Generative Frame Model
`PlayWorld`	PlayWorld: Learning Robot World Models from Autonomous Play
`WorldCache`	WorldCache: Accelerating World Models for Free via Heterogeneous Token
`DreamWorld`	DreamWorld: Unified World Modeling in Video Generation
`Olaf-World`	Olaf-World: Orienting Latent Actions for Video World Modeling
`EgoWM`	Walk Through Paintings : Ego-centric World Models from Internet Priors
`ReWorld`	ReWorld: Multi-Dimensional Reward Modeling for Embodied World Models
`NeoVerse`	NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos
`GrndCtrl`	GrndCtrl: Grounding World Models via Self-Supervised Reward Alignment
`C^3`	World Models That Know When They Don’t Know: Controllable Video Generation with Calibrated Uncertainty
`GAIA-3`	GAIA-3: Scaling World Models to Power Safety and Evaluation
`WristWorld`	WristWorld: Generating Wrist-Views via 4D World Models for Robotic Manipulation
`WorldSplat`	WorldSplat: Gaussian-Centric Feed-Forward 4D Scene Generation for Autonomous Driving
`WorldForge`	WorldForge: Unlocking Emergent 3D/4D Generation in Video Diffusion Model via Training-Free Guidance
`PEWM`	Learning Primitive Embodied World Models: Towards Scalable Robotic Learning
`Video Policy`	Video Generators are Robot Policies
`Ego-PM`	Ego-centric Predictive Model Conditioned on Hand Trajectories
`GWM`	GWM: Towards Scalable Gaussian World Models for Robotic Manipulation
`M3arsSynth`	Martian World Models: Controllable Video Synthesis with Physically Accurate 3D Reconstructions
`AirScape`	AirScape: An Aerial Generative World Model with Motion Controllability
`robot4dgen`	Geometry-aware 4D Video Generation for Robot Manipulation
`Genesis`	Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistency
`MinD`	MinD: Learning A Dual-System World Model for Real-Time Planning and Implicit Risk Analysis
`RealPlay`	PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling
`COME`	COME: Adding Scene-Centric Forecasting Control to Occupancy World Model
`MeWM`	Medical World Model: Generative Simulation of Tumor Evolution for Treatment Planning
`StateSpaceDiffuser`	StateSpaceDiffuser: Bringing Long Context to Diffusion World Models
`GeoDrive`	GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Control
`3DPEWM`	Learning 3d persistent embodied world models
`RoboTransfer`	RoboTransfer: Geometry-Consistent Video Diffusion for Robotic Visual Policy Transfer
`FlowDreamer`	FlowDreamer: A RGB-D World Model with Flow-based Motion Representations for Robot Manipulation
`LaDi-WM`	LaDi-WM: A Latent Diffusion-based World Model for Predictive Manipulation
`LangToMo`	Pixel motion as universal representation for robot control
`DreamGen`	DreamGen: Unlocking Generalization in Robot Learning through Video World Models
`Learning to Drive`	Learning to Drive from a World Model
`Tesseract`	Tesseract: Learning 4d embodied world models
`UWM`	Unified world models: Coupling video and action diffusion for pretraining on large robotic datasets
`ViMo`	ViMo: A Generative Visual GUI World Model for App Agents
`DiST-4D`	DiST-4D: Disentangled Spatiotemporal Diffusion with Metric Depth for 4D Driving Scene Generation
`Aether`	Aether: Geometric-aware unified world modeling
`Cosmos transfer1`	Cosmos transfer1: Conditional world generation with adaptive multimodal control
`GAIA-2`	GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving
`EDELINE`	EDELINE: Enhancing Memory in Diffusion-based World Models via Linear-Time Sequence Modeling
`MaskGWM`	MaskGWM: A Generalizable Driving World Model with Video Mask Reconstruction
`Dreamdrive`	Dreamdrive: Generative 4d scene modeling from street view images
`Cosmos`	Cosmos World Foundation Model Platform for Physical AI
`VPP`	Video prediction policy: A generalist robot policy with predictive visual representations
`Imagine-2-drive`	Imagine-2-drive: High-fidelity world modeling in carla for autonomous vehicles
`GenEx`	Generative World Explorer
`DOME`	DOME: Taming Diffusion Model into High-Fidelity Controllable Occupancy World Model
`Panacea+`	Panacea+: Panoramic and Controllable Video Generation for Autonomous Driving
`Bevworld`	Bevworld: A multimodal world model for autonomous driving via unified bev latent space
`Delphi`	Unleashing generalization of end-to-end autonomous driving with controllable long video generation
`Occsora`	Occsora: 4d occupancy generation models as world simulators for autonomous driving
`GenAD`	Generalized Predictive Model for Autonomous Driving
`WorldGPT`	Worldgpt: a sora-inspired video ai agent as rich world models from text and image inputs
`DriveDreamer-2`	DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation
`WoVoGen`	WoVoGen: World Volume-Aware Diffusion for Controllable Multi-Camera Driving Scene Generation
`Drive-WM`	Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving
`Panacea`	Panacea: Panoramic and controllable video generation for autonomous driving
`Drivingdiffusion`	DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion model
`DriveDreamer`	DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving
`UniPi`	Learning universal policies via text-guided video generation

Autoregressive Diffusion

Design	Paper	Link

`MemLearner`	MemLearner: Learning to Query Context memory for Video World Models
`ActWorld`	ActWorld: From Explorable to Interactive World Model via Action-Aware Memory
`DreamX-World`	DreamX-World 1.0: A General-Purpose Interactive World Model
`Mirage`	Latent Spatial Memory for Video World Models
`Echo-Memory`	Echo-Memory: A Controlled Study of Memory in Action World Models
`Cosmos 3`	Cosmos 3: Omnimodal World Models for Physical AI
`Gamma-World`	Gamma-World: Generative Multi-Agent World Modeling Beyond Two Players
`SANA-WM`	SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformer
`Matrix-Game 3.0`	Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory
`WorldCam`	WorldCam: Interactive Autoregressive 3D Gaming Worlds with Camera Pose as a Unifying Geometric Representation
`SWM`	Grounding World Simulation Models in a Real-World Metropolis
`LIVE`	LIVE: Long-horizon Interactive Video World Modeling
`LingBot-World`	Advancing Open-source World Models
`UniDrive-WM`	UniDrive-WM: Unified Understanding, Planning and Generation World Model For Autonomous Driving
`Yume1.5`	Yume1.5: A Text-Controlled Interactive World Generation Model
`HY-World 1.5`	HY-World 1.5: A Systematic Framework for Interactive World Modeling with Real-Time Latency and Geometric Consistency
`Astra`	Astra: General Interactive World Model with Autoregressive Denoising
`RELIC`	RELIC: Interactive Video World Model with Long-Horizon Memory
`ANWM`	Aerial World Model for Long-horizon Visual Generation and Navigation in 3D Space
`WorldPack`	WorldPack: Compressed Memory Improves Spatial Consistency in Video World Modeling
`Hunyuan-GameCraft-2`	Hunyuan-GameCraft-2: Instruction-following Interactive Game World Model
`PAN`	PAN: A World Model for General, Interactable, and Long-Horizon World Simulation
`Emu3.5`	Emu3.5: Native Multimodal Models are World Learners
`OmniNWM`	OmniNWM: Omniscient Driving Navigation World Models
`WoW`	WoW: Towards a World omniscient World model Through Embodied Interaction
`LongScape`	PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling
`Dreamer V4`	Training Agents Inside of Scalable World Models
`Genie Envisioner`	Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulation
`LiDARCrafter`	LiDARCrafter: Dynamic 4D World Modeling from LiDAR Sequences
`Yan`	Yan: Foundational Interactive Video Generation
`Matrix-Game 2.0`	Matrix-Game 2.0: An Open-Source, Real-Time, and Streaming Interactive World Model
`Yume`	Yume: An Interactive World Generation Model
`Geometry Forcing`	Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modeling
`spmem`	Video World Models with Long-term Spatial Memory
`STAGE`	STAGE: A Stream-Centric Generative World Model for Long-Horizon Driving-Scene Simulation
`CaM`	Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrieval
`DeepVerse`	DeepVerse: 4D Autoregressive Video Generation as a World Model
`Epona`	Epona: Autoregressive Diffusion World Model for Autonomous Driving
`SceneDiffuser++`	SceneDiffuser++: City-Scale Traffic Simulation via a Generative World Model
`VMem`	VMem: Consistent Interactive Video Scene Generation with Surfel-Indexed View Memory
`Hunyuan-GameCraft`	Hunyuan-GameCraft: High-dynamic Interactive Game Video Generation with Hybrid History Condition
`Matrix-Game`	Matrix-Game: Interactive World Foundation Model
`PEVA`	Whole-Body Conditioned Egocentric Video Prediction
`NFD`	Playing with Transformer at 30+ FPS via Next-Frame Diffusion
`VRAG`	Learning World Models for Interactive Video Generation
`DriVerse`	DriVerse: Navigation World Model for Driving Simulation via Multimodal Trajectory Prompting and Motion Alignment
`WORLDMEM`	WORLDMEM: Long-term Consistent World Simulation with Memory
`Adaworld`	Adaworld: Learning adaptable world models with latent actions
`TSWM`	Toward Stable World Models: Measuring and Addressing World Instability in Generative Environments
`Gamefactory`	Gamefactory: Creating new games with generative interactive videos
`PlayGen`	Playable Game Generation
`InfinityDrive`	InfinityDrive: Breaking Time Limits in Driving World Models
`GEM`	Gem: A generalizable ego-vision multimodal world model for fine-grained ego-motion, object dynamics, and scene composition control
`UniMLVG`	UniMLVG: Unified Framework for Multi-view Long Video Generation with Comprehensive Control Capabilities for Autonomous Driving
`The Matrix`	The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control
`NWM`	Navigation World Models
`Copilot4D`	Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion
`Drivingsphere`	Drivingsphere: Building a high-fidelity 4d world for closed-loop simulation
`Gamegen-x`	Gamegen-x: Interactive open-world game video generation
`Genie 2`	Genie 2: A large-scale foundation world model
`oasis`	Oasis: A Universe in a Transformer
`GameNGen`	Diffusion models are real-time game engines
`IRAsim`	IRASim: A Fine-Grained World Model for Robot Manipulation
`diamond`	Diffusion for World Modeling: Visual Details Matter in Atari
`Vista`	Vista: A generalizable driving world model with high fidelity and versatile controllability
`UniSim`	UniSim: Learning Interactive Real-World Simulators

1.3. Embedding Prediction

JEPA

Design	Paper	Link
`AdaJEPA`	AdaJEPA: An Adaptive Latent World Model
`FR3D`	Future Dynamic 3D Reconstruction: A 3D World Model with Disentangled Ego-Motion
`Sub-JEPA`	Sub-JEPA: Subspace Gaussian Regularization for Stable End-to-End World Models
`HWM`	Hierarchical Planning with Latent World Models
`LeWorldModel`	LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels
`V-JEPA 2.1`	V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning
`Temporal Straightening`	Temporal Straightening for Latent Planning
`C-JEPA`	Causal-JEPA: Learning World Models through Object-Level Latent Interventions
`VLA-JEPA`	VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Model
`DDP-WM`	DDP-WM: Disentangled Dynamics Prediction for Efficient World Models
`DINO-world`	Back to the Features: DINO as a Foundation for Video World Models
`V-JEPA 2`	V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
`SIVWM`	Sparse Imagination for Efficient Visual World Model Planning
`seq-JEPA`	seq-JEPA: Autoregressive Predictive Learning of Invariant-Equivariant World Models
`Flare`	Flare: Robot learning with implicit world modeling
`OSVI-WM`	OSVI-WM: One-Shot Visual Imitation for Unseen Tasks using World-Model-Guided Trajectory Generation
`EchoWorld`	EchoWorld: Learning Motion-Aware World Models for Echocardiography Probe Guidance
`AD-L-JEPA`	AD-L-JEPA: Self-Supervised Spatial World Models with Joint Embedding Predictive Architecture for Autonomous Driving with LiDAR Data
`DINO-Foresight`	DINO-Foresight: Looking into the Future with DINO
`DINO-WM`	DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
`LAW`	Enhancing End-to-End Autonomous Driving with Latent World Model
`V-JEPA`	Revisiting Feature Prediction for Learning Visual Representations from Video
`IWM`	Learning and Leveraging World Models in Visual Representation Learning

1.4. State Transition

Latent State-Space Modeling

Design	Paper	Link

`NE-Dreamer`	Next Embedding Prediction Makes World Models Stronger
`BOOM`	Bootstrap Off-policy with World Model
`GASv2`	Visuomotor Grasping with World Models for Surgical Robots
`LPS`	Latent Policy Steering with Embodiment-Agnostic Pretrained World Models
`EMERALD`	Accurate and Efficient World Modeling with Masked Latent Transformers
`FOUNDER`	FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Making
`NavMorph`	NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environments
`Raw2Drive`	Raw2Drive: Reinforcement Learning with Aligned World Models for End-to-End Autonomous Driving
`SSVWM`	Long-Context State-Space Video World Models
`PIN-WM`	PIN-WM: Learning Physics-Informed World Models for Non-Prehensile Manipulation
`ReDRAW`	Adapting World Models with Latent-State Dynamics Residuals
`WoTE`	End-to-End Driving with Online Trajectory Evaluation via BEV World Model
`DyWA`	DyWA: Dynamics-Adaptive World Action Model for Generalizable Non-Prehensile Manipulation
`DMWM`	DMWM: Dual-Mind World Model with Long-Term Imagination
`Simulus`	Uncovering Untapped Potential in Sample-Efficient World Model Agents
`S5WM`	Accelerating Model-Based Reinforcement Learning with State-Space World Models
`AdaWM`	AdaWM: Adaptive World Model based Planning for Autonomous Driving
`RoboHorizon`	RoboHorizon: An LLM-Assisted Multi-View World Model for Long-Horizon Robotic Manipulation
`LS-Imagine`	Open-World Reinforcement Learning over Long Short-Term Imagination
`-`	Mitigating Covariate Shift in Imitation Learning for Autonomous Vehicles Using Latent Space Generative World Models
`GenRL`	GenRL: Multimodal-Foundation World Models for Generalization in Embodied Agents
`DriveWorld`	DriveWorld: 4D Pre-Trained Scene Understanding via World Models for Autonomous Driving
`Puppeteer`	Hierarchical World Models as Visual Whole-Body Humanoid Controllers
`R2I`	Mastering Memory Tasks with World Models
`REM`	Improving Token-Based World Models with Parallel Observation Prediction
`Think2Drive`	Think2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving
`MUVO`	MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representations
`STORM`	STORM: Efficient Stochastic Transformer based World Models for Reinforcement Learning
`HarmonyDream`	HarmonyDream: Task Harmonization Inside World Models
`TD-MPC2`	TD-MPC2: Scalable, Robust World Models for Continuous Control
`MoDem-V2`	MoDem-V2: Visuo-Motor World Models for Real-World Robot Manipulation
`SWIM`	Structured World Models from Human Videos
`Dynalang`	Learning to Model the World with Language
`SafeDreamer`	SafeDreamer: Safe Reinforcement Learning with World Models
`CoWorld`	Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learning
`TWM`	Transformer-Based World Models Are Happy With 100k Interactions
`MV-MWM`	Multi-View Masked World Models for Visual Robotic Manipulation
`DreamV3`	Mastering Diverse Domains through World Models
`IRIS`	Transformers are Sample-Efficient World Models
`MWM`	Masked World Models for Visual Control
`DayDreamer`	DayDreamer: World Models for Physical Robot Learning
`CADDY`	Playable Video Generation
`DreamV2`	Mastering Atari with Discrete World Models
`DreamV1`	Dream to Control: Learning Behaviors by Latent Imagination
`PlaNet`	Learning Latent Dynamics for Planning from Pixels

Object-Centric Modeling

Design	Paper	Link

`LPWM`	Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modeling
`FIOC-WM`	Learning Interactive World Model for Object-Centric Reinforcement Learning
`Dyn-o`	Dyn-O: Building Structured World Models with Object-Centric Representations
`SlotPi`	SlotPi: Physics-informed Object-centric Reasoning Models
`-`	Object-Centric World Model for Language-Guided Manipulation
`DisWM`	Disentangled World Models: Learning to Transfer Semantic Knowledge from Distracting Videos for Reinforcement Learning
`Objects matter`	Objects matter: object-centric world models improve reinforcement learning in visually complex environments
`Dreamweaver`	Dreamweaver: Learning Compositional World Models from Pixels
`MEAD`	Efficient Exploration and Discriminative World Model Learning with an Object-Centric Abstraction
`slotSSM`	Slot State Space Models
`RoboDreamer`	RoboDreamer: Learning Compositional World Models for Robot Imagination
`SSWM`	Slot Structured World Models
`cosmos`	Neurosymbolic Grounding for Compositional World Models
`FOCUS`	FOCUS: Object-Centric World Models for Robotics Manipulation
`SlotFormer`	SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models
`HOWM`	Toward Compositional Generalization in Object-Oriented World Modeling
`G-SWM`	Improving Generative Imagination in Object-Centric World Models

1.5. Other Architectures

Design	Paper	Link

`PhysWorld`	PhysWorld: From Real Videos to World Models of Deformable Objects via Physics-Aware Demonstration Synthesis
`FASTopoWM`	FASTopoWM: Fast-Slow Lane Segment Topology Reasoning with Latent World Models
`GWM`	Graph World Model
`World4Drive`	World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Model
`Orbis`	Orbis: Overcoming Challenges of Long-Horizon Prediction in Driving World Models
`LiDARWM`	Towards foundational LiDAR world models with efficient latent flow matching
`ManiGaussian++`	ManiGaussian++: General Robotic Bimanual Manipulation with Hierarchical Gaussian World Model
`GAF`	GAF: Gaussian Action Field as a 4D Representation for Dynamic World Modeling in Robotic Manipulation
`HWM`	Humanoid World Models: Open World Foundation Models for Humanoid Robotics
`DriveX`	DriveX: Omni Scene Modeling for Learning Generalizable World Knowledge in Autonomous Driving
`OccProphet`	OccProphet: Pushing Efficiency Frontier of Camera-Only 4D Occupancy Forecasting with Observer-Forecaster-Refiner Framework
`FleetWM`	Multi-Task Interactive Robot Fleet Learning with Visual World Models
`GaussianWorld`	GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Prediction
`NeMo`	Neural volumetric world models for autonomous driving

2. Datasets & Benchmarks

2.1. Foundational World Modeling

General World Prediction and Simulation

Design	Paper	Link

`MemoBench`	MemoBench: Benchmarking World Modeling in Dynamically Changing Environments
`WRBench`	Current World Models Lack a Persistent State Core
`MBench`	MBench: A Comprehensive Benchmark on Memory Capability for Video World Models
`WBench`	WBench: A Comprehensive Multi-turn Benchmark for Interactive Video World Model Evaluation
`iWorld-Bench`	iWorld-Bench: A Benchmark for Interactive World Models with a Unified Action Generation Framework
`WR-Arena`	World Reasoning Arena
`CoW-Bench`	The Trinity of Consistency as a Defining Principle for General World Models
`MIND`	MIND: Benchmarking Memory Consistency and Action Control in World Models
`DynamicVerse`	DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modeling
`4DWorldBench`	4DWorldBench: A Comprehensive Evaluation Framework for 3D/4D World Generation Models
`Gen-ViRe`	Can World Simulators Reason? Gen-ViRe: A Generative Visual Reasoning Benchmark
`World-in-World`	World-in-World: World Models in a Closed-Loop World
`OmniWorld`	OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modeling
`Sekai`	Sekai: A Video Dataset towards World Exploration
`WorldPrediction`	WorldPrediction: A Benchmark for High-level World Modeling and Long-horizon Procedural Planning
`WorldScore`	WorldScore: A Unified Evaluation Benchmark for World Generation
`MM-OR`	MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environments
`WorldModelBench`	WorldModelBench: Judging Video Generation Models As World Models
`OpenHumanVid`	OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generation
`EgoVid-5M`	EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generation
`Ego-Exo4D`	Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
`InternVid`	InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation
`Ego4D`	Ego4D: Around the World in 3,000 Hours of Egocentric Video
`WebVid-2M`	Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval
`EPIC-KITCHENS-100`	Rescaling Egocentric Vision
`HowTo100M`	HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
`COIN`	COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis
`SSV2`	The "something something" video database for learning and evaluating visual common sense
`Kinetics`	The Kinetics Human Action Video Dataset
`YouTube-8M`	YouTube-8M: A Large-Scale Video Classification Benchmark
`UCF101`	UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild

Physics and Causality Benchmark

Design	Paper	Link

`PhysEditWorld`	PhysEditWorld: A Large-Scale Dataset Toward Physics-Editable World Models
`Omni-WorldBench`	Omni-WorldBench: Towards a Comprehensive Interaction-Centric Evaluation for World Models
`WorldBench`	WorldBench: Disambiguating Physics for Diagnostic Evaluation of World Models
`VideoVerse`	VideoVerse: How Far is Your T2V Generator from a World Model?
`PAI-Bench`	Physical ai bench: A comprehensive benchmark for physical ai generation and understanding
`PhysVidBench`	Can Your Model Separate Yolks with a Water Bottle? Benchmarking Physical Commonsense Understanding in Video Generation Models
`PBench`	PBench: A benchmark for evaluating generative models
`IntPhys 2`	IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environments
`T2VPhysBench`	T2VPhysBench: A First-Principles Benchmark for Physical Consistency in Text-to-Video Generation
`PisaBench`	PISA Experiments: Exploring Physics Post-Training for Video Diffusion Models by Watching Stuff Drop
`WISA-32K`	WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generation
`VideoPhy-2`	VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation
`VBench-2.0`	VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness
`PhyCoBench`	A Physical Coherence Benchmark for Evaluating Video Generation Models via Optical Flow-guided Frame Prediction
`Physics-IQ`	Do generative video models understand physical principles?
`PhyGenBench`	Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation
`PhyBench`	PhyBench: A Physical Commonsense Benchmark for Evaluating Text-to-Image Models
`VideoPhy`	VideoPhy: Evaluating Physical Commonsense for Video Generation
`Physion++`	Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Properties
`InfLevel`	Benchmarking Progress to Infant-Level Physical Reasoning in AI
`VoE`	A Benchmark for Modeling Violation-of-Expectation in Physical Reasoning Across Event Categories
`Physion`	Physion: Evaluating Physical Prediction from Vision in Humans and Machines
`CoPhy`	CoPhy: Counterfactual Learning of Physical Dynamics
`IntPhys`	IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning

2.2. Domain-specific World Modeling

Embodied AI and Robotics

Design	Paper	Link

`WMBench (GigaWorld-1)`	GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation
`WorldArena 2.0`	WorldArena 2.0: Extending Embodied World Model Benchmarking on Modality, Functionality and Platform
`GE-Sim 2.0`	Genie Envisioner World Simulator 2.0
`RoboWM-Bench`	RoboWM-Bench: A Benchmark for Evaluating World Models in Robotic Manipulation
`WorldArena`	WorldArena: A Unified Benchmark for Evaluating Perception and Functional Utility of Embodied World Models
`DreamDojo`	DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos
`WoW-World-Eval`	Wow, wo, val! A Comprehensive Embodied World Model Evaluation Turing Test
`GigaWorld-0`	GigaWorld-0: World Models as Data Engine to Empower Embodied AI
`Target-Bench`	Target-Bench: Can World Models Achieve Mapless Path Planning with Semantic Targets
`WoW`	WoW: Towards a world omniscient world model through embodied interaction
`Meta-World+`	Meta-World+: An improved, standardized, rl benchmark
`AgiBot-World`	AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems
`RoboCasa`	RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots
`DROID`	DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
`BEHAVIOR-1K`	BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation
`MimicGen`	MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations
`OXE`	Open X-Embodiment: Robotic Learning Datasets and RT-X Models
`RH20T`	RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot
`VP²`	A Control-Centric Benchmark for Video Prediction
`RT-1`	RT-1: Robotics Transformer for Real-World Control at Scale
`BC-Z`	BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning
`CALVIN`	CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks
`BridgeData V2`	BridgeData V2: A Dataset for Robot Learning at Scale
`LIBERO`	LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning
`Isaac Gym`	Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning
`RoboNet`	RoboNet: Large-Scale Multi-Robot Learning
`Meta-World`	Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
`RLBench`	RLBench: The Robot Learning Benchmark & Learning Environment

Autonomous Driving

Design	Paper	Link

`WorldLens`	WorldLens: Full-Spectrum Evaluations of Driving World Models in Real World
`ACT-Bench`	ACT-Bench: Towards Action Controllable World Models for Autonomous Driving
`DrivingDojo`	DrivingDojo Dataset: Advancing Interactive and Knowledge-Enriched Driving World Model
`DriveArena`	DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Driving
`NAVSIM`	NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking
`CarDreamer`	CarDreamer: Open-Source Learning Platform for World Model based Autonomous Driving
`OpenDV-2K`	GenAD: Generalized Predictive Model for Autonomous Driving
`ZOD`	Zenseact Open Dataset: A Large-Scale and Diverse Multimodal Dataset for Autonomous Driving
`Occ3D`	Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving
`OpenOccupancy`	OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perception
`Argoverse 2`	Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting
`KITTI-360`	KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D
`NuPlan`	NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles
`ONCE`	One Million Scenes for Autonomous Driving: ONCE Dataset
`WOMD`	Large Scale Interactive Motion Forecasting for Autonomous Driving: The Waymo Open Motion Dataset
`Lyft Level 5`	One Thousand and One Hours: Self-driving Motion Prediction Dataset
`A2D2`	A2D2: Audi Autonomous Driving Dataset
`Waymo`	Scalability in Perception for Autonomous Driving: Waymo Open Dataset
`Argoverse`	Argoverse: 3D Tracking and Forecasting With Rich Maps
`INTERACTION`	INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps
`SemanticKITTI`	SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences
`nuScenes`	nuScenes: A Multimodal Dataset for Autonomous Driving
`BDD100K`	BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning
`ApolloScape`	The ApolloScape Dataset for Autonomous Driving
`CARLA`	CARLA: An Open Urban Driving Simulator
`KITTI`	Are we ready for autonomous driving? The KITTI vision benchmark suite

Interactive Environments and Gaming

Design	Paper	Link

`SCOPE`	SCOPE: Simulating Cross-game Operations in Playable Environments for FPS World Models
`WorldMark`	WorldMark: A Unified Benchmark Suite for Interactive Video World Models
`WildWorld`	WildWorld: A Large-Scale Dataset for Dynamic World Modeling with Actions and Explicit State toward Generative ARPG
`Matrix-Game-MC`	Matrix-Game: Interactive World Foundation Model
`LOOPNAV`	Toward Memory-Aided World Models: Benchmarking via Spatial Consistency
`JARVIS-VLA`	JARVIS-VLA: Post-Training Large-Scale Visual Language Models to Play Visual Games with Keyboards and Mouse
`GF-Minecraft`	GameFactory: Creating New Games with Generative Interactive Videos
`The Matrix`	The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control
`GameGen-X`	GameGen-X: Interactive Open-world Game Video Generation
`Mars`	Mars: Situated Inductive Reasoning in an Open-World Environment
`Crafter`	Benchmarking the Spectrum of Agent Capabilities
`CS-Deathmatch`	Counter-Strike Deathmatch with Large-Scale Behavioural Cloning
`MineRL`	The MineRL 2020 Competition on Sample Efficient Reinforcement Learning using Human Priors
`NLE`	The NetHack Learning Environment
`Procgen`	Leveraging Procedural Generation to Benchmark Reinforcement Learning
`DMC`	DeepMind Control Suite
`ALE`	The Arcade Learning Environment: An Evaluation Platform for General Agents

3. Others

Survey

Paper	Link

World Model for Robot Learning: A Comprehensive Survey
OpenWorldLib: A Unified Codebase and Definition of Advanced World Models
Video Generation Models as World Models: Efficient Paradigms, Architectures and Algorithms
Learning to Model the World: A Survey of World Models in Artificial Intelligence
Towards Generalist Embodied AI: A Survey on World Models for VLA Agents
Video Generation Models in Robotics -- Applications, Research Challenges, Future Directions
Digital Twin AI: Opportunities and Challenges from Large Language Models to World Models
Progressive Robustness-Aware World Models in Autonomous Driving: A Review and Outlook
Simulating the Visual World with Artificial Intelligence: A Roadmap
A Step Toward World Models: A Survey on Robotic Manipulation
The Safety Challenge of World Models for Embodied AI Agents: A Review
A Comprehensive Survey on World Models for Embodied AI
3D and 4D World Modeling: A Survey
Large Model Empowered Embodied AI: A Survey on Decision-Making and Embodied Learning
A Survey: Learning Embodied Intelligence from Physical Simulators and World Models
From 2D to 3D Cognition: A Brief Survey of General World Models
A Survey on World Models Grounded in Acoustic Physical Information
World Models in Artificial Intelligence: Sensing, Learning, and Reasoning Like a Child
The Role of World Models in Shaping Autonomous Driving: A Comprehensive Survey
A Survey of World Models for Autonomous Driving
Understanding World or Predicting Future? A Comprehensive Survey of World Models
Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyond
World Models for Autonomous Driving: An Initial Survey
A Path Towards Autonomous Machine Intelligence

GitHub Repo

This part overlaps somewhat with the Survey section.

Repo	Link

WMFactory 0.5: One environment · One procedure · Eleven interactive world models
A minimalist repository for training video world models based on diffusion-forcing
Awesome Video World Models with AR Diffusion
Awesome From Video Generation to World Model
A Curated List of Amazing Works in World Modeling, spanning applications in Embodied AI, Autonomous Driving, Natural Laguage Processing and Agents
A curated list of papers for World Models for General Video Generation, Embodied AI, and Autonomous Driving
World Models for Autonomous Driving (and Robotic) papers
Awesome 3D and 4D World Models
Simulating the Real World: Survey & Resources
A curated list of world models for autonomous driving
A curated list of resources related to World Models for Autonomous Driving (WMAD)
A Survey: Learning Embodied Intelligence from Physical Simulators and World Models
A Comprehensive Survey on World Models for Embodied AI
A Survey on World Models Grounded in Acoustic Physical Information
A Comprehensive Survey on General World Models and Beyond
Understanding World or Predicting Future? A Comprehensive Survey of World Models
From Masks to Worlds: A Hitchhiker's Guide to World Models

Workshop

Venue	Workshop	Link

`ECCV 2026`	How to Build Effective World Models for Embodied AI
`CVPR 2026`	GigaBrain Challenge 2026: World Models & VLA for Embodied Intelligence
`CVPR 2026`	Video World Models
`CVPR 2026`	World Models Meet Active Sensing and Closed-Loop Planning
`ICLR 2026`	Workshop on World Models: Understanding, Modelling and Scaling
`NeurIPS 2025`	Workshop on Bridging Language, Agent, and World Models for Reasoning and Planning
`NeurIPS 2025`	Embodied World Models for Decision Making
`ICCV 2025`	Reliable and Interactable World Models: Geometry, Physics, Interactivity and Real-World Generalization
`ICML 2025`	Building Physically Plausible World Models
`CVPR 2025`	Benchmarking World Models
`ICLR 2025`	World Models: Understanding, Modelling, and Scaling

Theory

Design	Paper	Link

`Orca`	Orca: The World is in Your Mind
`LoopWM`	Looped World Models
`OrbiSim`	OrbiSim: World Models as Differentiable Physics Engines for Embodied Intelligence
`-`	Physically Native World Models: A Hamiltonian Perspective on Generative World Modeling
`ZWM`	Zero-shot World Models Are Developmentally Efficient Learners
`-`	Human Cognition in Machines: A Unified Perspective of World Models
`EB-JEPA`	A Lightweight Library for Energy-Based Joint-Embedding Predictive Architectures
`-`	Research on World Models Is Not Merely Injecting World Knowledge into Specific Tasks
`-`	What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?
`-`	Closing the Train-Test Gap in World Models for Gradient-Based Planning
`WWM`	Web World Model
`DWM`	Dexterous World Models	[](https://github.com/s

Truncated — view the full README on GitHub.

Similar repos

JiahuaDong/Awesome-World-Models

353

ziqihuangg/Awesome-From-Video-Generation-to-World-Model

A list of works on video generation towards world model

518

liujiuming123/Awesome-Interactive-World-Model

A Comprehensive Survey of Interactive Video World Models

234

knightnemo/Awesome-World-Models

A Curated List of Awesome Works in World Modeling, Aiming to Serve as a One-stop Resource for…

3,441

NTUMARS/Awesome-World-Model-for-Robotics-Policy

811

ALEEEHU/World-Simulator

[IEEE TPAMI 2026] Simulating the Real World: Survey & Resources, which contains our survey…

389

yueen-ma/Awesome-VLA

671

HaoranZhuExplorer/World-Models-Autonomous-Driving-Survey

A curated list of world models for autonomous driving.

535

Contributors

33 commits

2 commits

2 commits

1 commits

AIWorldLab/Awesome-Vision-World-Model

README

Awesome Vision World Models

:newspaper: News

Table of Contents

1. Designs

1.1. Sequential Generation

Visual Autoregressive Modeling

MLLM-guided Multimodal Autoregressive Model

1.2. Diffusion-based Generation

Latent Diffusion

Autoregressive Diffusion

1.3. Embedding Prediction

JEPA

1.4. State Transition

Latent State-Space Modeling

Object-Centric Modeling

1.5. Other Architectures

2. Datasets & Benchmarks

2.1. Foundational World Modeling

General World Prediction and Simulation

Physics and Causality Benchmark

2.2. Domain-specific World Modeling

Embodied AI and Robotics

Autonomous Driving

Interactive Environments and Gaming

3. Others

Survey

GitHub Repo

Workshop

Theory

Contributors

Similar repos

AIWorldLab/Awesome-Vision-World-Model

README

Awesome Vision World Models

:newspaper: News

Table of Contents

1. Designs

1.1. Sequential Generation

Visual Autoregressive Modeling

MLLM-guided Multimodal Autoregressive Model

1.2. Diffusion-based Generation

Latent Diffusion

Autoregressive Diffusion

1.3. Embedding Prediction

JEPA

1.4. State Transition

Latent State-Space Modeling

Object-Centric Modeling

1.5. Other Architectures

2. Datasets & Benchmarks

2.1. Foundational World Modeling

General World Prediction and Simulation

Physics and Causality Benchmark

2.2. Domain-specific World Modeling

Embodied AI and Robotics

Autonomous Driving

Interactive Environments and Gaming

3. Others

Survey

GitHub Repo

Workshop

Theory

Similar repos

Contributors