Introduction

As part of OmniGen2, we introduce a new benchmark for in-context generation, OmniContext, which aims to provide a more comprehensive evaluation of models' in-context generation abilities. It incorporates a diverse set of input images and instructions, and utilizes GPT-4.1 for interpretable, metric-driven assessment.

Project Page: https://vectorspacelab.github.io/OmniGen2

Github Repo for OmniContext: https://github.com/VectorSpaceLab/OmniGen2

Arxiv Paper for OmniContext: https://arxiv.org/abs/2506.18871

How to use it ?

Loading Data

from datasets import load_dataset
data = load_dataset('OmniGen2/OmniContext', dataset, split='train')

Data Format

All data in OmniContext are standardized to the following format:

{
    "task_type": "The task type of this sample",
    "instruction": "User instruction for in-context image generation",
    "input_images": "A list of all input images",
    "instruction_langugae": "The language used in the instruction",
    "key": "A unique identifier for the data sample. The generated image should be saved as {key}.png for evaluation",
}

Evaluation

This repository provides test data for OmniContext. If you wish to use this dataset for automated evaluation, please refer to our github.

Task statistics

Citation

@article{wu2025omnigen2,
  title={OmniGen2: Exploration to Advanced Multimodal Generation},
  author={Chenyuan Wu and Pengfei Zheng and Ruiran Yan and Shitao Xiao and Xin Luo and Yueze Wang and Wanli Li and Xiyan Jiang and Yexin Liu and Junjie Zhou and Ze Liu and Ziyi Xia and Chaofan Li and Haoge Deng and Jiahao Wang and Kun Luo and Bo Zhang and Defu Lian and Xinlong Wang and Zhongyuan Wang and Tiejun Huang and Zheng Liu},
  journal={arXiv preprint arXiv:2506.18871},
  year={2025}
}