Generating Intermediate Representations for Compositional Text-To-Image Generation

About

For this project, we investigated the use of intermediate representations (e.g. segmentation maps and depth maps) to enhance the performance of Stable Diffusion model. More specifically, we operated in two stages, in the first stage using Stable Diffusion to generate an intermediate representation(s), and in the second stage using ControlNet to produce an image based on these representation(s).

Qualitative results.

Quantitative results.

The pipeline of our model. In the first step, we used Stable Diffusion to generate single or multiple aligned representation(s). In the second step, we used these representation(s) to generate the final image.

The pipeline of our alignment model. To align two representations, two U-Nets of pre-trained Stable Diffusion models were interleaved by adding temporal layers.

Acknowledgments

We used the following resources in our research:
LanguageBind - https://github.com/PKU-YuanGroup/LanguageBind
Vision and Language Transformer (VITL) - https://github.com/dandelin/ViLT
Depth Anything - https://github.com/LiheYoung/Depth-Anything
ControlNet - https://github.com/lllyasviel/ControlNet, https://huggingface.co/thibaud/controlnet-sd21