PRICE: Prediction of Real-world Interactions with Constraints Evaluation

PRICE-V0.1 is a benchmark for instruction-conditional image-to-image generation grounded in real-world robot, egocentric, and third-person interaction data.

Given an initial-state image and a natural-language instruction, a model predicts the target-state image resulting from the instructed action. The benchmark evaluates instruction following, scene consistency, and physical plausibility.

The evaluation code and protocol live in evaluation/image_gen/PRICE/ in orca-wm/Orca.

Benchmark overview

Each PRICE example pairs a natural-language instruction and an initial-state image with a reference target-state image. The examples below illustrate the benchmark's diverse robot and egocentric interaction scenes, viewpoints, and action outcomes.

Examples from PRICE. Each pair shows the initial state and the target state corresponding to the instruction.

Dataset structure

The dataset contains one test split with 100 samples and the following principal columns:

Additional provenance fields, timestamps, indices, random seed, and image checksums are retained from the source manifest.

Usage

Load PRICE directly from Hugging Face:

from datasets import load_dataset

dataset = load_dataset("BAAI/PRICE", split="test")
sample = dataset[0]

initial_image = sample["query"]
target_image = sample["output"]
instruction = sample["lang"]

Load the prepared folder locally:

from datasets import load_dataset

dataset = load_dataset("imagefolder", data_dir="data", split="test")

Evaluation

Use the evaluator and judge prompts from the GitHub repository. Model predictions should be named by the corresponding id, following the layout documented there.

Data sources

PRICE-V0.1 contains selected samples derived from AgiBot World, HomeInteract, PE-Video, and PsiBot SynData. Consult the PRICE paper and GitHub repository for complete source citations and selection details.

License

PRICE is released under the Apache License 2.0.

Version

This layout corresponds to PRICE-V0.1. Tag the Hugging Face Dataset repository with v0.1 and pin that revision from the evaluation code for reproducible results.