Abstract

Diffusion models for garment-centric human generation from text or image prompts have garnered emerging attention for their great application potential. However, existing methods often face a dilemma: lightweight approaches, such as adapters, are prone to generate inconsistent textures; while finetune-based methods involve high training costs and struggle to maintain the generalization capabilities of pretrained diffusion models, limiting their performance across diverse scenarios. To address these challenges, we propose DreamFit, which incorporates a lightweight Anything-Dressing Encoder specifically tailored for the garment-centric human generation.

DreamFit has three key advantages:

1. Lightweight training: with the proposed adaptive attention and LoRA modules, DreamFit significantly minimizes the model complexity to 83.4M trainable parameters.
2. Anything-Dressing: Our model generalizes surprisingly well to a wide range of (non-)garments, creative styles, and prompt instructions, consistently delivering high-quality results across diverse scenarios.
3. Plug-and-play: DreamFit is engineered for smooth integration with any community control plugins for diffusion models, ensuring easy compatibility and minimizing adoption barriers.

To further enhance generation quality, DreamFit leverages pretrained large multi-modal models (LMMs) to enrich the prompt with fine-grained garment descriptions, thereby reducing the prompt gap between training and inference. We conduct comprehensive experiments on both 768 × 512 high-resolution benchmarks and in-the-wild images. DreamFit surpasses all existing methods, highlighting its state-of-the-art capabilities of garment-centric human generation.

Overview

Our method constructs an Anything-Dressing Encoder utilizing LoRA layers. The reference image features are extracted by the Anything-Dressing Encoder and then passed into the denoising UNet via adaptive attention.

Furthermore, we incorporate Large Multimodal Models (LMM) into the inference process to reduce the text prompt gap between the training and testing.

Installation Guide

1. Clone our repo:

git clone https://github.com/bytedance/DreamFit.git

2. Create new virtual environment:

conda create -n dreamfit python==3.10
conda activate dreamfit

3. Install our dependencies by running the following command:

pip install -r requirements.txt
pip install flash-attn --no-build-isolation --use-pep517 

Models

1. You can download the pretrained models Here. Download the checkpoint to pretrained_models folder.
2. If you want to inference with StableDiffusion1.5 version, you need to download the stable-diffusion-v1-5, sd-vae-ft-mse to pretrained_models. If you want to generate images of different styles, you can download the corresponding stylized model, such as RealisticVision, to pretrained_models.
3. If you want to inference with Flux version, you need to download the flux-dev to pretrained_models folder
4. If you want to inference with pose control, you need to download the Annotators to pretrained_models folder

The folder structures should look like these:

├── pretrained_models/
|   ├── flux_i2i_with_pose.bin
│   ├── flux_i2i.bin
│   ├── flux_tryon.bin
│   ├── sd15_i2i.ckpt
|   ├── stable-diffusion-v1-5/
|   |   ├── ...
|   ├── sd-vae-ft-mse/
|   |   ├── diffusion_pytorch_model.bin
|   |   ├── ...
|   ├── Realistic_Vision_V6.0_B1_noVAE(or other stylized model)/
|   |   ├── unet/
|   |   |   ├── diffusion_pytorch_model.bin
|   |   |   ├── ...
|   |   ├── ...
|   ├── Annotators/
|   |   ├── body_pose_model.pth
|   |   ├── facenet.pth
|   |   ├── hand_pose_model.pth
|   ├── FLUX.1-dev/
|   |   ├── flux1-dev.safetensors
|   |   ├── ae.safetensors
|   |   ├── tokenizer
|   |   ├── tokenizer_2
|   |   ├── text_encoder
|   |   ├── text_encoder_2
|   |   ├── ...

Inference

Garment-Centric Generation

# inference with FLUX version
bash run_inference_dreamfit_flux_i2i.sh \
    --cloth_path example/cloth/cloth_1.png \
    --image_text "A woman wearing a white Bape T-shirt with a colorful ape graphic and bold text." \
    --save_dir "." \
    --seed 164143088151

# inference with StableDiffusion1.5 version
bash run_inference_dreamfit_sd15_i2i.sh \
    --cloth_path example/cloth/cloth_3.jpg\
    --image_text "A woman with curly hair wears a pink t-shirt with a logo and white stripes on the sleeves, paired with white trousers, against a plain white background."\
    --ref_scale 1.0 \
    --base_model pretrained_models/Realistic_Vision_V6.0_B1_noVAE/unet/diffusion_pytorch_model.bin \
    --base_model_load_method diffusers \
    --save_dir "." \
    --seed 28

Tips:

1. If you have multiple pieces of clothing, you can splice them onto one picture, as shown in the second row.
2. Use --help to check the meaning of each argument.

Garment-Centric Generation with Pose Control

bash run_inference_dreamfit_flux_i2i_with_pose.sh \
    --cloth_path example/cloth/cloth_1.png \
    --pose_path example/pose/pose_1.jpg \
    --image_text "A woman wearing a white Bape T-shirt with a colorful ape graphic and bold text." \
    --save_dir "." \
    --seed 16414308815

Tryon

bash run_inference_dreamfit_flux_tryon.sh \
    --cloth_path example/cloth/cloth_1.png \
    --keep_image_path example/tryon/keep_image_4.png \
    --image_text "A woman wearing a white Bape T-shirt with a colorful ape graphic and bold text and a blue jeans." \
    --save_dir "." \
    --seed 16414308815

Tips:

1. Keep image is obtained by drawing the openpose on the garment-agnostic region.
2. The generation code for keep image cannot be open-sourced for the time being. As an alternative, we have provided several keep images for testing.

Disclaimer

Most images used in this repository are sourced from the Internet. These images are solely intended to demonstrate the capabilities of our research. If you have any concerns, please contact us, and we will promptly remove any inappropriate content.

This project aims to make a positive impact on the field of AI-driven image generation. Users are free to create images using this tool, but they must comply with local laws and use it responsibly. The developers do not assume any responsibility for potential misuse by users.

Citation

@article{lin2024dreamfit,
  title={DreamFit: Garment-Centric Human Generation via a Lightweight Anything-Dressing Encoder},
  author={Lin, Ente and Zhang, Xujie and Zhao, Fuwei and Luo, Yuxuan and Dong, Xin and Zeng, Long and Liang, Xiaodan},
  journal={arXiv preprint arXiv:2412.17644},
  year={2024}
}

Acknowledgements

Thanks to x-flux and Moore-AnimateAnyone repositories, for their open research and exploration.

Contact

If you have any comments or questions, please open a new issue or feel free to contact Ente Lin and Xin Dong.