Awesome Instructional Editing

A Survey of Instruction-Guided Image and Media Editing in LLM Era

A collection of academic articles, published methodology, and datasets on the subject of Instruction-Guided Image and Media Editing.

• Instruction-Guided Editing Controls for Images and Multimedia: A Survey in LLM era
  • Existing Surveys
  • Taxonomy
  • Approaches for Image Editing
  • Approaches for Media Editing
  • Datasets
    • Type: General
    • Type: Image Captioning
    • Type: ClipArt
    • Type: VQA
    • Type: Semantic Segmentation
    • Type: Object Classification
    • Type: Depth Estimation
    • Type: Aesthetic-Based Editing
    • Type: Dialog-Based Editing
  • Evaluation Metrics

A sortable version is available here: https://awesome-instruction-editing.github.io/

Please read and cite our paper:

Nguyen, Thanh Tam, Zhao Ren, Trinh Pham, Phi Le Nguyen, Quoc Viet Hung Nguyen, and Hongzhi Yin. "A review of instruction-guided image editing." Engineering Applications of Artificial Intelligence 163 (2026): 112953. https://doi.org/10.1016/j.engappai.2025.112953

🔖 News!!!

📌 We are actively tracking the latest research and welcome contributions to our repository and survey paper. If your studies are relevant, please feel free to create an issue or a pull request.

📰 2026-02-28: Our website is live at https://awesome-instruction-editing.github.io/. Thanks a lot for your citations.

📰 2025-10-30: Our work has been published in Engineering Applications of Artificial Intelligence. Thanks a lot for your continuous supports.

📰 2024-11-15: Our paper Instruction-Guided Editing Controls for Images and Multimedia: A Survey in LLM era has been revised into version 1 with new methods and dicussions.

🔍 Citation

If you find this work helpful in your research, welcome to cite the paper and give a ⭐.

@article{nguyen2026review,
  title={A review of instruction-guided image editing},
  author={Nguyen, Thanh Tam and Ren, Zhao and Pham, Trinh and Le Nguyen, Phi and Nguyen, Quoc Viet Hung and Yin, Hongzhi},
  journal={Engineering Applications of Artificial Intelligence},
  volume={163},
  pages={112953},
  year={2026},
  publisher={Elsevier}
}

@article{nguyen2024instruction,
  title={Instruction-Guided Editing Controls for Images and Multimedia: A Survey in LLM era},
  author={Thanh Tam Nguyen and Zhao Ren and Trinh Pham and Thanh Trung Huynh and Phi Le Nguyen and Hongzhi Yin and Quoc Viet Hung Nguyen},
  journal={arXiv preprint arXiv:2411.09955},
  year={2024}
}

Existing Surveys

Pipeline

Approaches for Image Editing

Approaches for Media Editing

Datasets

Type: General

Type: Image Captioning

Type: ClipArt

Type: VQA

Type: Semantic Segmentation

Type: Object Classification

Type: Depth Estimation

Type: Aesthetic-Based Editing

Type: Dialog-Based Editing

Evaluation Metrics

Benchmark Results

Notes:

• Scaling caveats: iEdit reports CLIPScore (%) and SSIM-M (% on edited/background regions). PIE-Bench reports “CLIP Semantics” (whole/edited) as un-normalized cosine-like scores (~20–26). LOCATEdit shows SSIM as ×10² and LPIPS unscaled (values ~39–42). Forgedit’s CLIPScore is cosine (0–1).
• [a] RegionDrag reports LPIPS×100; we divide by 100 (thus 9.9→0.099, 9.2→0.092).

Experiment Configuration

pip install --upgrade diffusers transformers accelerate safetensors torch torchvision

import torch, PIL.Image as Image
from diffusers import (
    StableDiffusionInstructPix2PixPipeline,
    StableDiffusionDiffEditPipeline,
    PaintByExamplePipeline,
    DDIMScheduler, DDIMInverseScheduler,
)

DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

# 1) InstructPix2Pix — text-guided global/local edits
def run_ip2p(input_path, instruction, out_path="out_ip2p.png", steps=30):
    pipe = StableDiffusionInstructPix2PixPipeline.from_pretrained(
        "timbrooks/instruct-pix2pix", torch_dtype=torch.float16 if DEVICE=="cuda" else torch.float32
    ).to(DEVICE)
    image = Image.open(input_path).convert("RGB")
    result = pipe(prompt=instruction, image=image,
                  num_inference_steps=steps, guidance_scale=7.5, image_guidance_scale=1.5).images[0]
    result.save(out_path); return out_path

# 2) DiffEdit — automatic mask + latent inversion for targeted edits
def run_diffedit(input_path, source_prompt, target_prompt, out_path="out_diffedit.png", steps=50):
    init_img = Image.open(input_path).convert("RGB")
    pipe = StableDiffusionDiffEditPipeline.from_pretrained(
        "runwayml/stable-diffusion-v1-5",
        torch_dtype=torch.float16 if DEVICE=="cuda" else torch.float32
    ).to(DEVICE)
    pipe.scheduler = DDIMScheduler.from_config(pipe.scheduler.config)
    pipe.inverse_scheduler = DDIMInverseScheduler.from_config(pipe.scheduler.config)

    mask = pipe.generate_mask(image=init_img, source_prompt=source_prompt, target_prompt=target_prompt,
                              num_inference_steps=steps)[0]
    inv = pipe.invert(prompt=source_prompt, image=init_img, num_inference_steps=steps)
    edited = pipe(prompt=target_prompt, negative_prompt=source_prompt,
                  image=init_img, mask_image=mask, latents=inv.latents,
                  num_inference_steps=steps).images[0]
    edited.save(out_path); return out_path

# 3) Paint-by-Example — exemplar-guided local replacement
def run_pbe(input_path, mask_path, example_path, out_path="out_pbe.png", steps=50):
    pipe = PaintByExamplePipeline.from_pretrained(
        "Fantasy-Studio/Paint-by-Example",
        torch_dtype=torch.float16 if DEVICE=="cuda" else torch.float32
    ).to(DEVICE)
    image = Image.open(input_path).convert("RGB")
    mask  = Image.open(mask_path).convert("L")         # white = repaint
    ex    = Image.open(example_path).convert("RGB")
    edited = pipe(image=image, mask_image=mask, example_image=ex,
                  num_inference_steps=steps, guidance_scale=5.0).images[0]
    edited.save(out_path); return out_path

# Example usage (where input.jpg, fruit.jpg, etc. are your input data)
run_ip2p("input.jpg", "make the sky pink at sunset")
run_diffedit("fruit.jpg", "a bowl of apples", "a bowl of pears")
run_pbe("room.jpg", "room_mask.png", "new_chair.jpg")

Repo-based Configuration

Add-it (training-free insertion)

git clone https://github.com/NVlabs/addit && cd addit
conda env create -f environment.yml && conda activate addit
# real image insertion:
python run_CLI_addit_real.py \
  --source_image "images/bed_dark_room.jpg" \
  --prompt_source "A photo of a bed in a dark room" \
  --prompt_target "A photo of a dog lying on a bed in a dark room" \
  --subject_token "dog"

FreeEdit (mask-free, reference-based)

git clone https://github.com/hrz2000/FreeEdit && cd FreeEdit
pip install -r requirements.txt
# check README / demo notebook for single-command inference

Grounded-Instruct-Pix2Pix (auto target grounding)

git clone https://github.com/arthur-71/Grounded-Instruct-Pix2Pix && cd Grounded-Instruct-Pix2Pix
pip install -r requirements.txt && python -m spacy download en_core_web_sm
# install GroundingDINO (per README), then open the provided notebook:
# jupyter notebook grounded-instruct-pix2pix.ipynb

RegionDrag (fast region edits/UI)

git clone https://github.com/Visual-AI/RegionDrag && cd RegionDrag
pip install -r requirements.txt
# see UI_GUIDE.md for the GUI runner; a minimal script is included in the repo

ZONE (localized editing)

git clone https://github.com/lsl001006/ZONE && cd ZONE
pip install -r requirements.txt
python demo.py --input your.jpg --prompt "make the mug red" --mask path/to/mask.png

D-Edit (freestyle mask-conditioned)

git clone https://github.com/collovlab/d-edit && cd d-edit
pip install -r requirements.txt
python app.py --input your.jpg --mask mask.png --prompt "replace the sofa with a blue one"

Pix2Pix-Zero / On-the-Fly (training-free, edit direction)

git clone https://github.com/pix2pixzero/pix2pix-zero && cd pix2pix-zero
pip install -r requirements.txt
# see README + HF demo link for quick usage

python src/edit_synthetic.py \
    --results_folder "output/synth_editing" \
    --prompt_str "a high resolution painting of a cat in the style of van gogh" \
    --task "cat2dog"

Disclaimer

Feel free to contact us if you have any queries or exciting news. In addition, we welcome all researchers to contribute to this repository and further contribute to the knowledge of this field.

If you have some other related references, please feel free to create a Github issue with the paper information. We will glady update the repos according to your suggestions. (You can also create pull requests, but it might take some time for us to do the merge)

Backup Statistics