VP-Bench

VP-Bench is the official evaluation benchmark for FlowInOne. It is a rigorously curated benchmark assessing instruction faithfulness, spatial precision, visual realism, and content consistency across eight distinct visual prompting tasks.

Evaluation

Our evaluation scripts are now available on GitHub!

Dataset Subsets

The dataset contains 8 subsets, each corresponding to a distinct visual instruction task:

Dataset Features

• input_image (image): The input visual prompt image (with rendered instruction).
• output_image (image): The ground-truth output image.
• recognized_text (string): The text instruction rendered in the input image (extracted via OCR annotation).
• subset (string): The subset name.
• category (string): Sub-category within a subset (empty string if not applicable).
• image_name (string): The image filename.
• input_relpath (string): Relative path of the input image within the subset.
• output_relpath (string): Relative path of the output image within the subset.
• pair_id (string): Stable SHA1 identifier for each input-output pair.

Loading the Dataset

# class2image
from datasets import load_dataset
ds = load_dataset("CSU-JPG/VPBench", "class2image", split="train")
# text2image
from datasets import load_dataset
ds = load_dataset("CSU-JPG/VPBench", "text2image", split="train")
# text_in_image
from datasets import load_dataset
ds = load_dataset("CSU-JPG/VPBench", "text_in_image", split="train")
# force
from datasets import load_dataset
ds = load_dataset("CSU-JPG/VPBench", "force", split="train")
# text_box_control
from datasets import load_dataset
ds = load_dataset("CSU-JPG/VPBench", "text_box_control", split="train")
# trajectory
from datasets import load_dataset
ds = load_dataset("CSU-JPG/VPBench", "trajectory", split="train")
# vismarker
from datasets import load_dataset
ds = load_dataset("CSU-JPG/VPBench", "vismarker", split="train")
# doodles
from datasets import load_dataset
ds = load_dataset("CSU-JPG/VPBench", "doodles", split="train")
# Load All Subsets
from datasets import load_dataset, concatenate_datasets
subsets = ["class2image", "text2image", "text_in_image", "force",
           "text_box_control", "trajectory", "vismarker", "doodles"]
ds_all = concatenate_datasets([
    load_dataset("CSU-JPG/VPBench", name=s, split="train") for s in subsets
])

Evaluation Results

We evaluate multiple methods on VP-Bench using three state-of-the-art VLM evaluators (Gemini3, GPT-5.2, Qwen3.5) and human judges. The metric is success ratio (higher is better). Total denotes the average success rate across all eight task categories.

Abbreviations: C2I: class-to-image · T2I: text-to-image · TIE: text-in-image edit · FU: force understanding · TBE: text & bbox edit · TU: trajectory understanding · VME: visual marker edit · DE: doodles edit

Evaluator: Gemini3

Evaluator: GPT-5.2

Evaluator: Qwen3.5

Evaluator: Human

Citation

If you found our work useful, please consider citing:

@article{yi2026flowinoneunifyingmultimodalgenerationimagein,
      title={FlowInOne:Unifying Multimodal Generation as Image-in, Image-out Flow Matching}, 
      author={Junchao Yi and Rui Zhao and Jiahao Tang and Weixian Lei and Linjie Li and Qisheng Su and Zhengyuan Yang and Lijuan Wang and Xiaofeng Zhu and Alex Jinpeng Wang},    
      journal={arXiv preprint arXiv:2604.06757},
      year={2026}
}