MuLAn: : A Multi Layer Annotated Dataset for Controllable Text-to-Image Generation

MuLAn is a novel dataset comprising over 44K MUlti-Layer ANnotations of RGB images as multilayer, instance-wise RGBA decompositions, and over 100K instance images. It is composed of MuLAn-COCO and MuLAn-LAION sub-datasets, which contain a variety of image decompositions in terms of style, composition and complexity. With MuLAn, we provide the first photorealistic resource providing instance decomposition and occlusion information for high quality images, opening up new avenues for text-to-image generative AI research. With this, we aim to encourage the development of novel generation and editing technology, in particular layer-wise solutions.

FAQ:

1. The LVIS dataset is equivalent to COCO 2017 dataset so please make sure you have the correct version of the dataset. Furthermore, the images come from both the traning and validation splits so check both subfolders of the original dataset.
2. The LAION dataset is based on LAION Aesthetic v2 6.5+. Given that img2dataset library outputs different indices based on which images have been download we released the links of the images we have used in this dataset. Those can be found in the file_id_url_linking.csv and need to be downloaded prior to following the Dataset decomposition steps.

Dataset format

In order to respect the base datasets' LICENCEs we have releasead MuLAn in annotation format.

Each image is associated with a pickle file structured as below. We have also released a small script that given a csv with the base image/annotation pairs will automatically reconstruct the decomposed images and save the captioning and path metadata in a separate csv.

"captioning": {
    "llava": LLaVa model details
    "blip2": BLIP 2 model details
    "clip": CLIP model details
}
"background": {
    "llava": Detailed background LLaVa caption
    "blip2": COCO style BLIP 2 background caption chosen by CLIP
    "original_image_mask": Original image background content mask
    "inpainted_delta": Additive inpainted background content
}
"image": {
    "llava": Detailed original image LLaVa caption
    "blip2": COCO style BLIP 2 original image  caption chosen by CLIP.
}
"instances": {
    "blip2": COCO style BLIP 2 instance caption chosen by CLIP.
    "original_image_mask": Original image instance content mask
    "inpainted_delta": Additive inpainted instance content
    "instance_alpha": Alpha layer of the inpainted instance
}

Dataset decomposition

First you need to make sure you have the unrar package for ubuntu. You can install it by using the following command.

sudo apt-get install rar unrar

Then the command below will extract the dataset.

unrar x -e mulan.part001.rar

Afterwards create the required conda environment

conda env create --name mulan --file=mulan_env.yml
conda activate mulan

Then manually create a csv with two column image and annotation similarly with the toy example below. Please pay attention to COCO dataset specifically as some base images are from the train2017 subset some are from the val2017 one.

image, annotation
<path_to_image>/<image_id>.jpg, <path_to_annotation>/<image_id>.p.zl
<path_to_image>/<image_id>.jpg, <path_to_annotation>/<image_id>.p.zl
<path_to_image>/<image_id>.jpg, <path_to_annotation>/<image_id>.p.zl

We advise to create to separate csvs, one for the COCO dataset and one for the LAION Aesthetic V2 6.5 in order to guarantee no image id clashes.

The provided script can then be used to reconstruct the RGBA stacks. Please be advised that we are using joblib to paralelise the decomposition so your CPU and I/O might be heavily impacted during the script running.

Be careful of the following:

• output_path needs to be without the trailing /
• number_of_processes if unspecified will default to 2 * number of cores

python3 dataset_decomposition.py \
    --csv_path='/path/to/images/and/annotations/file.csv' \
    --output_path='/path/to/where/images/will/be/decomposed' \
    --number_of_processes=<<number of cores>>

In the /path/to/where/images/will/be/decomposed, the script will generate multiple images per original RGB image following the structure below as well as a meta_data.csv file. The csv will have three columns inside paths of the individual layers, blip2 caption of the layer and llava caption of the same layer. The llava caption will be N/A for instances as we have not generate those.

<<image_id>>-layer_0.png - Background RGB Image
<<image_id>>-layer_x.png - Instance X RGBA Image

Examples

COCO

LAION Aesthetic v2 6.5

Possible applications

Instance Addition through MuLAn finetuned InstructPix2Pix

Instance Generation through MuLAn finetuned StableDiffusion v1.5

Reference

Please do not forget to cite our work if you are using this dataset in your research.

Corresponding author is Petru-Daniel Tudosiu (petru.daniel.tudosiu@huawei.com).

@InProceedings{Tudosiu_2024_CVPR,
  author    = {Tudosiu, Petru-Daniel and Yang, Yongxin and Zhang, Shifeng and Chen, Fei and McDonagh, Steven and Lampouras, Gerasimos and Iacobacci, Ignacio and Parisot, Sarah},
  title     = {MULAN: A Multi Layer Annotated Dataset for Controllable Text-to-Image Generation},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month     = {June},
  year      = {2024},
  pages     = {22413-22422}
}