EditReward-Bench: A Human-Aligned Reward Model Benchmark for Instruction-Guided Image Editing

This repository contains EditReward-Bench, a new benchmark introduced in the paper EditReward: A Human-Aligned Reward Model for Instruction-Guided Image Editing.

Introduction

Recent advances in image editing with natural language instructions have shown promising progress, particularly with closed-source models. However, open-source models often lag due to the lack of a reliable reward model for scaling up high-quality synthetic training data. To address this critical bottleneck, the paper introduces EditReward, a human-aligned reward model trained with a new large-scale human preference dataset called EditReward-Data (over 200K preference pairs).

To rigorously assess EditReward and future models, we also introduce EditReward-Bench, a new benchmark built upon high-quality annotations, which includes more difficult multi-way preference prediction tasks. EditReward demonstrates superior alignment with human preferences in instruction-guided image editing tasks, achieving state-of-the-art human correlation on established benchmarks and on EditReward-Bench itself.

• Paper: https://huggingface.co/papers/2509.26346
• Project Page: https://tiger-ai-lab.github.io/EditReward
• Code: https://github.com/TIGER-AI-Lab/EditReward

Dataset Statistics

• Total unique instructions: 431
• Score range: 2.0 - 8.0 (Overall score = Instruction Following + Visual Quality)
• Average score difference: 2.23
• Total images: 1,739 (all embedded in parquet files)

Download EditReward-Bench

You can download the EditReward-Bench dataset using the Hugging Face CLI:

huggingface-cli download --repo-type dataset TIGER-Lab/EditReward-Bench --local-dir /your-local-dataset-path

Usage

Please refer to the github code for evaluation! The EditReward-bench's evaluation is available at EditReward/evaluate.

Benchmark Results: Alignment with Humans

The following tables present the experimental results from the paper, showcasing various models' alignment with human preferences on established benchmarks and on EditReward-Bench.

Experimental Results: Alignment with Humans

Method	GenAI-Bench	AURORA-Bench	ImagenHub	EditReward-Bench (Overall)
Random	25.90	33.43	--	13.84
Human-to-Human	--	--	41.84	--
Proprietary Models
GPT-4o	53.54	50.81	38.21	28.31
GPT-5	59.61	47.27	40.85	37.81
Gemini-2.0-Flash	53.32	44.31	23.69	33.47
Gemini-2.5-Flash	57.01	47.63	41.62	38.02
Open-Source VLMs
Qwen2.5-VL-3B-Inst	42.76	30.69	-2.54	26.86
Qwen2.5-VL-7B-Inst	40.48	38.62	18.59	29.75
Qwen2.5-VL-32B-Inst	39.28	37.06	26.87	28.72
MiMo-VL-7B-SFT-2508	57.89	30.43	22.14	31.19
ADIEE	59.96	55.56	34.50	--
Reward Models (Ours)
EditReward (on Qwen2.5-VL-7B)	63.97	59.50	36.18	36.78
EditReward (on MiMo-VL-7B)	65.72	63.62	35.20	38.42

---

EditReward-Bench Results (K-way preference prediction)

Method	EditReward-Bench (K=2)	EditReward-Bench (K=3)	EditReward-Bench (K=4)	EditReward-Bench (Overall)
Random	25.81	11.33	1.35	13.84
Human-to-Human	--	--	--	--
Proprietary Models
GPT-4o	45.69	27.33	7.31	28.31
GPT-5	57.53	38.51	12.84	37.81
Gemini-2.0-Flash	52.43	33.33	13.51	33.47
Gemini-2.5-Flash	58.61	39.86	12.16	38.02
Open-Source VLMs
Qwen2.5-VL-3B-Inst	51.07	20.27	2.71	26.86
Qwen2.5-VL-7B-Inst	52.69	24.67	3.38	29.75
Qwen2.5-VL-32B-Inst	50.54	25.27	4.05	28.72
MiMo-VL-7B-SFT-2508	49.46	30.41	9.46	31.19
ADIEE	--	--	--	--
Reward Models (Ours)
EditReward (on Qwen2.5-VL-7B)	56.99	36.00	10.81	36.78
EditReward (on MiMo-VL-7B)	56.45	42.67	11.49	38.42

Citation

Please kindly cite our paper if you use our code, data, models or results:

@article{wu2025editreward,
  title={EditReward: A Human-Aligned Reward Model for Instruction-Guided Image Editing},
  author={Wu, Keming and Jiang, Sicong and Ku, Max and Nie, Ping and Liu, Minghao and Chen, Wenhu},
  journal={arXiv preprint arXiv:2509.26346},
  year={2025}
}