Diffusers API of Transparent Image Layer Diffusion using Latent Transparency

Create transparent image with Diffusers! Please check the Github repo here: https://github.com/rootonchair/diffuser_layerdiffuse

This is a port to Diffuser from original SD Webui's Layer Diffusion to extend the ability to generate transparent image with your favorite API

Paper: Transparent Image Layer Diffusion using Latent Transparency

What's new

• Added Diffusers-ready SDXL LayerDiffuse conditional weights for foreground-to-blending, background-to-blending, foreground-and-blend-to-background, and background-and-blend-to-foreground workflows.
• New remote weights: diffuser_layer_xl_fg2ble.safetensors, diffuser_layer_xl_bg2ble.safetensors, diffuser_layer_xl_fgble2bg.safetensors, and diffuser_layer_xl_bgble2fg.safetensors.
• The GitHub examples load these weights from this Hugging Face repo through the local HF cache by default.
• SDXL Forge weight conversion is now consolidated in scripts/convert_xl_layerdiffuse.py with --mode fg2ble|bg2ble|fgble2bg|bgble2fg.
• Demo scripts now expose CLI options for model, prompt, seed, output path, --variant, and --cpu-offload; run any script with --help for details.

Quickstart

Generate transparent image with SD1.5 models. In this example, we will use digiplay/Juggernaut_final as the base model

from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
import torch

from diffusers import StableDiffusionPipeline

from models import TransparentVAEDecoder
from loaders import load_lora_to_unet



if __name__ == "__main__":

    model_path = hf_hub_download(
        'LayerDiffusion/layerdiffusion-v1',
        'layer_sd15_vae_transparent_decoder.safetensors',
    )

    vae_transparent_decoder = TransparentVAEDecoder.from_pretrained("digiplay/Juggernaut_final", subfolder="vae", torch_dtype=torch.float16).to("cuda")
    vae_transparent_decoder.set_transparent_decoder(load_file(model_path))

    pipeline = StableDiffusionPipeline.from_pretrained("digiplay/Juggernaut_final", vae=vae_transparent_decoder, torch_dtype=torch.float16, safety_checker=None).to("cuda")

    model_path = hf_hub_download(
        'LayerDiffusion/layerdiffusion-v1',
        'layer_sd15_transparent_attn.safetensors'
    )

    load_lora_to_unet(pipeline.unet, model_path, frames=1)
    
    image = pipeline(prompt="a dog sitting in room, high quality", 
                       width=512, height=512,
                       num_images_per_prompt=1, return_dict=False)[0]