📌 Release

[03/2025] Code and paper are publicly available.

📑 Abstract

TL;DR: Conceptrol is a free lunch that elicits the personalized ability of zero-shot adapter by transforming image condition to visual specification contrained by textual concept, even outperforming fine-tuning methods.

CLICK for the full abstract

Personalized image generation with text-to-image diffusion models generates unseen images based on reference image content. Zero-shot adapter methods such as IP-Adapter and OminiControl are especially interesting because they do not require test-time fine-tuning. However, they struggle to balance preserving personalized content and adherence to the text prompt. We identify a critical design flaw resulting in this performance gap: current adapters inadequately integrate personalization images with the textual descriptions. The generated images, therefore, replicate the personalized content rather than adhere to the text prompt instructions. Yet the base text-to-image has strong conceptual understanding capabilities that can be leveraged.

We propose Conceptrol, a simple yet effective framework that enhances zero-shot adapters without adding computational overhead. Conceptrol constrains the attention of visual specification with a textual concept mask that improves subject-driven generation capabilities. It achieves as much as 89% improvement on personalization benchmarks over the vanilla IP-Adapter and can even outperform fine-tuning approaches such as Dreambooth LoRA. The source code will be released upon acceptance.

🚗 Quick Start

1. Environment Setup

conda create -n conceptrol python=3.10
conda activate conceptrol
pip install -r requirements.txt

2. Go to demo_sd.ipynb / demo_sdxl.ipynb / demo_flux.py for fun!

🛳️ Local Setup using Gradio

1. Start Gradio Interface

pip install gradio
gradio gradio_src/app.py

2. Use the GUI for fun!

📝 Supporting Models

✒️Citation

If you found this repository/our paper useful, please consider citing:

@article{he2025conceptrol,
  title={Conceptrol: Concept Control of Zero-shot Personalized Image Generation},
  author={Qiyuan He and Angela Yao},
  journal={arXiv preprint arXiv:2503.06568},
  year={2025}
}

❤️ Acknowledgement

We thank the following repositories for their great work:

diffusers, transformers, IP-Adapter, OminiControl