πArxiv | πProject Page | π»Github | πDataset | πHF Paper
We introduce Complex-Edit, a comprehensive benchmark designed to systematically evaluate instruction-based image editing models across instructions of varying complexity. To develop this benchmark, we harness GPT-4o to automatically collect a diverse set of editing instructions at scale.
Our approach follows a well-structured βChain-of-Editβ pipeline: we first generate individual atomic editing tasks independently and then integrate them to form cohesive, complex instructions. Additionally, we introduce a suite of metrics to assess various aspects of editing performance, along with a VLM-based auto-evaluation pipeline that supports large-scale assessments.
Our benchmark yields several notable insights:
The dataset is organized as follows:
βββ README.md
βββ test
βββ real
| βββ images
| | βββ 0000.png
| | βββ 0001.png
| | βββ 0002.png
β β βββ ...
β βββ metadata.jsonl
βββ syn
βββ images
βββ 0000.png
βββ 0001.png
βββ 0002.png
βββ metadata.jsonl
Input images are stored in test and instructions are stored in edit. real and syn refer to real-life input and synthetic input respectively.
The JSON file structure is demonstrated as below:
{
"reasoning": "...",
"original_sequence": [
{
"name": "Change Background",
"instruction": "Replace the existing background with a busy metropolitan skyline."
},
{
"name": "Add Special Effects",
"instruction": "Add motion blur to the cars to depict motion."
},
...
],
"sequence": [
{
"name": "Change Background",
"instruction": "Replace the existing background with a busy metropolitan skyline."
},
{
"name": "Add Special Effects",
"instruction": "Add motion blur to the cars."
},
...
],
"compound": [
{
"reasoning": "none",
"compound_instruction": "Replace the existing background with a busy metropolitan skyline."
},
{
"reasoning": "...",
"compound_instruction": "Replace the background with a busy metropolitan skyline and apply motion blur to the cars to simulate movement."
},
...
]
}
Each JSON file in edit contains a sequence of atmoic instructions sequence and 8 compound instructions in compound for a corresponding input image. original_sequence is the sequence of atomic instructions without simplification.
The compound instructions are at different complexity levels ranging from $C_1$ to $C_8$ in an ascending order.
from datasets import load_dataset
dataset = load_dataset("UCSC-VLAA/Complex-Edit")
sample = dataset["test_real"][0]
# Print the compound instructions. Complexity from C1 to C8.
for i, compound in enumerate(sample["edit"]["compound"]):
print(f"C{i + 1} Instruction: {compound['compound_instruction']}")
# Print the atomic instruction sequence.
for i, compound in enumerate(sample["edit"]["sequence"]):
print(f"Step #{i + 1} Atomic Instruction: {compound['instruction']}")
# Show the input image.
sample["image"].show()
2 commits
πArxiv | πProject Page | π»Github | πDataset | πHF Paper
We introduce Complex-Edit, a comprehensive benchmark designed to systematically evaluate instruction-based image editing models across instructions of varying complexity. To develop this benchmark, we harness GPT-4o to automatically collect a diverse set of editing instructions at scale.
Our approach follows a well-structured βChain-of-Editβ pipeline: we first generate individual atomic editing tasks independently and then integrate them to form cohesive, complex instructions. Additionally, we introduce a suite of metrics to assess various aspects of editing performance, along with a VLM-based auto-evaluation pipeline that supports large-scale assessments.
Our benchmark yields several notable insights:
The dataset is organized as follows:
βββ README.md
βββ test
βββ real
| βββ images
| | βββ 0000.png
| | βββ 0001.png
| | βββ 0002.png
β β βββ ...
β βββ metadata.jsonl
βββ syn
βββ images
βββ 0000.png
βββ 0001.png
βββ 0002.png
βββ metadata.jsonl
Input images are stored in test and instructions are stored in edit. real and syn refer to real-life input and synthetic input respectively.
The JSON file structure is demonstrated as below:
{
"reasoning": "...",
"original_sequence": [
{
"name": "Change Background",
"instruction": "Replace the existing background with a busy metropolitan skyline."
},
{
"name": "Add Special Effects",
"instruction": "Add motion blur to the cars to depict motion."
},
...
],
"sequence": [
{
"name": "Change Background",
"instruction": "Replace the existing background with a busy metropolitan skyline."
},
{
"name": "Add Special Effects",
"instruction": "Add motion blur to the cars."
},
...
],
"compound": [
{
"reasoning": "none",
"compound_instruction": "Replace the existing background with a busy metropolitan skyline."
},
{
"reasoning": "...",
"compound_instruction": "Replace the background with a busy metropolitan skyline and apply motion blur to the cars to simulate movement."
},
...
]
}
Each JSON file in edit contains a sequence of atmoic instructions sequence and 8 compound instructions in compound for a corresponding input image. original_sequence is the sequence of atomic instructions without simplification.
The compound instructions are at different complexity levels ranging from $C_1$ to $C_8$ in an ascending order.
from datasets import load_dataset
dataset = load_dataset("UCSC-VLAA/Complex-Edit")
sample = dataset["test_real"][0]
# Print the compound instructions. Complexity from C1 to C8.
for i, compound in enumerate(sample["edit"]["compound"]):
print(f"C{i + 1} Instruction: {compound['compound_instruction']}")
# Print the atomic instruction sequence.
for i, compound in enumerate(sample["edit"]["sequence"]):
print(f"Step #{i + 1} Atomic Instruction: {compound['instruction']}")
# Show the input image.
sample["image"].show()
2 commits