WM-ABench: An Atomic Evaluation Benchmark of World Modeling abilities of Vision-Language Models

Paper: Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation

WM-ABench is a comprehensive benchmark that evaluates whether Vision-Language Models (VLMs) can truly understand and simulate physical world dynamics, or if they rely on shortcuts and pattern-matching. The benchmark covers 23 dimensions of world modeling across 6 physics simulators with over 100,000 evaluation instances.

🔗 Visit the Official Website for leaderboard, paper, and walkthrough video.

What Does WM-ABench Test?

The benchmark evaluates VLMs on two main capabilities:

🔍 Perception Tasks - Can the model understand what's happening?

• Visual attributes (color, shape, material)
• Spatial relationships and 3D positioning
• Object counting and quantities
• Temporal relationships
• Motion detection and trajectories

🔮 Prediction Tasks - Can the model predict what happens next?

• Physics simulation (dropping, sliding, collisions)
• Mechanistic agent actions (lifting, pushing objects)
• Transitive reasoning (sequential actions)
• Compositional actions (multiple things happening at once)

Quick Start

#!pip install --upgrade --force-reinstall datasets fsspec huggingface_hub
from datasets import load_dataset

# Load a 100-instance subset for quick evaluation
ds = load_dataset("maitrix-org/WM-ABench", "Compositionality_maniskill_lift_subset")

# Or load the full dataset
# ds = load_dataset("maitrix-org/WM-ABench", "Compositionality_maniskill_lift")

print(ds['test'][0]['prompt'])

Dataset Structure

Each task provides:

• Video frames showing the initial scene/entire sequence of motion
• Multiple choice questions with 4 possible outcomes
• Hard negative options that look similar but are physically incorrect
• Rich metadata about object properties and scene setup

The benchmark uses data from 6 simulation frameworks: ThreeDWorld, ManiSkill 2 & 3, Habitat Lab 2.0, Physion, and Carla.

Key Features

Avoiding Shortcuts: Unlike other benchmarks, WM-ABench includes carefully designed "hard negative" answer choices that prevent models from succeeding through visual similarity alone. Models must truly understand physics to get the right answer.

Atomic Evaluation: WM-ABench provides fine-grained analysis of specific world modeling capabilities.

Scale: Over 100,000 test instances across 23 different dimensions of world modeling.

Data Fields

Fields vary by task, but typically include:

• source: Video frames of the initial state
• prompt: The question asked to the model
• image_choices or choices: Possible answer options
• answer: Index or value of the correct answer
• params: Scene setup details

Usage Notes

• All configurations have only a test split
• Each full configuration has a _subset version with 100 instances for faster evaluation
• Ground truth labels are programmatically generated from simulator states
• Human evaluation baselines show moderate to high agreement (Fleiss' κ > 0.4)

Citation

@misc{gao2025visionlanguagemodelsinternalworld,
  title={Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation},
  author={Qiyue Gao and Xinyu Pi and Kevin Liu and Junrong Chen and Ruolan Yang and Xinqi Huang and Xinyu Fang and Lu Sun and Gautham Kishore and Bo Ai and Stone Tao and Mengyang Liu and Jiaxi Yang and Chao-Jung Lai and Chuanyang Jin and Jiannan Xiang and Benhao Huang and Zeming Chen and David Danks and Hao Su and Tianmin Shu and Ziqiao Ma and Lianhui Qin and Zhiting Hu},
  year={2025},
  eprint={2506.21876},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}

License: Apache 2.0