Stable Diffusion XL Inference Server

A high-performance SDXL image generation API built with FastAPI, optimized for AMD RDNA 4 GPUs. The server manages pipeline lifecycle, model caching, LoRA hot-loading, multi-ControlNet spatial conditioning, and prompt conditioning via Compel.

Features

• SDXL text-to-image and image-to-image generation
• Lightning mode — reduced-step inference (8 steps, low CFG) for near-real-time output
• LoRA support — load one or more LoRA adapters per request with independent scale control
• Multi-ControlNet — depth map and canny edge conditioning via Xinsir SDXL ControlNets
• IP-Adapter — style/reference image conditioning
• Divergent Spaces — heterogeneous per-image control (different depth/canny/mask per batch item)
• Inpainting — mask-guided inpainting via StableDiffusionXLControlNetInpaintPipeline
• ControlNet asset generation — on-server depth map (DA3), edge map (PiDiNet), and compositing mask extraction
• Node-based workflow engine — composable pipelines for hi-res refinement, tiling, upscaling, and spatial transforms
• Automatic model caching — first load converts single-file .safetensors to diffusers format for ~3× faster subsequent loads
• Pipeline warmup on startup — MIOpen, Triton, and TunableOp caches are populated at boot, eliminating cold-start latency
• Batched output — multiple images per request; single PNG returned when batch_size=1
• CORS-configurable — origin allowlist via environment variable

Requirements

Project Structure

├── main.py                  # FastAPI application, endpoints, lifespan
├── openapi.yaml             # OpenAPI 3.1 specification
├── requirements.txt
├── caches/
│   ├── warmed_configs.json  # Tracks which model/resolution combos are warmed
│   └── models/              # Diffusers-format model cache (auto-generated)
│       └── juggernaut/
└── src/
    ├── models.py            # Pydantic request/response schemas
    ├── pipeline.py          # Pipeline loading, caching, warmup, generation
    ├── loras.py             # LoRA loading and adapter management (/loras router)
    ├── prompt.py            # Prompt pre-processing and quality tags
    ├── controlnet.py        # ControlNet asset generation (/spatial-assets router)
    ├── executor.py          # DAG execution engine
    └── nodes/
        ├── base_node.py     # Abstract node base class
        ├── compel_node.py   # Prompt conditioning via Compel
        ├── text2image.py    # Text-to-image inference node
        ├── image2image.py   # Image-to-image inference node
        ├── hi_res_node.py   # Hi-res refinement pass node
        ├── tiling_node.py   # Tiling plan node
        ├── upscale_node.py  # Upscaling node
        ├── transform_node.py# Spatial transform node
        └── response_node.py # HTTP response packaging node

Setup

1. Install dependencies

pip install -r requirements.txt

2. Place models

Put SDXL .safetensors checkpoint files in ~/sd_models/. The filename (without extension) becomes the model identifier:

~/sd_models/juggernaut.safetensors
~/sd_models/pony.safetensors

On first load, each model is converted to diffusers format and cached under caches/models/<name>/ for faster subsequent starts.

3. Place LoRAs (optional)

Put LoRA .safetensors files in ~/sd_loras/:

~/sd_loras/detail_enhancer.safetensors

4. Run the server

uvicorn main:app --host 0.0.0.0 --port 8000

Environment Variables

API

All endpoints that accept image files use multipart/form-data.

---

POST /generate/image

Generate one or more images from a text prompt. Supports text-to-image, image-to-image, multi-ControlNet, IP-Adapter, and inpainting via Divergent Spaces.

Request (multipart/form-data):

Divergent Spaces ({n}.* fields): when any indexed field is present, each batch item receives its own set of spatial priors. The number of batch items must equal batch_size. Including a {n}.mask field switches the pipeline to StableDiffusionXLControlNetInpaintPipeline.

Response:

• image/png — when batch_size=1
• application/zip — when batch_size>1, containing image_0.png … image_N.png and metrics.json

// metrics.json
{
  "latency": 4.82,
  "throughput": 0.207,
  "breakdown": {
    "pipeline_load_time": 0.001,
    "lora_load_time": 0.0,
    "prompt_processing_time": 0.12,
    "generation_time": 4.7
  }
}

---

GET /models

List model identifiers available for generation (.safetensors stems from ~/sd_models/).

Response (application/json):

["juggernaut", "pony"]

---

GET /loras/

List LoRA adapter names available for use (.safetensors stems from ~/sd_loras/).

Response (application/json):

["detail_enhancer", "style_anime"]

---

POST /spatial-assets/generate

Generate ControlNet conditioning assets from a single input image: a DA3 depth map, a PiDiNet edge map, and five foreground/background mask pairs fused from both.

Request (multipart/form-data):

Response (application/zip) — controlnet_priors.zip containing:

Mask variants: conservative_baseline, conservative_depth_push, conservative_tight_edge, conservative_deep_smooth, conservative_surgical.

---

POST /workflows/

Full hi-res workflow: text-to-image (or image-to-image) → 4× upscale → tiling → hi-res refinement pass → spatial transform → response.

Request (multipart/form-data):

Response: image/png (single) or application/zip (batch).

---

POST /workflows/image/

Simple workflow: text-to-image or image-to-image only, no upscaling or hi-res pass. Accepts the same form fields as /workflows/ (except hires_strength is ignored).

Response: image/png (single) or application/zip (batch).

---

Architecture Notes

• Pipeline caching — Only one pipeline is held in VRAM at a time. Switching models triggers a full cleanup (gc.collect() + torch.cuda.empty_cache()) to avoid OOM.
• VAE — Uses madebyollin/sdxl-vae-fp16-fix with tiling and slicing enabled to keep peak VRAM low during decode.
• Schedulers — Standard mode: Euler Ancestral (30 steps, CFG 7.0). Lightning mode: DPM Solver Multistep (8 steps, CFG 1.5).
• Prompt enhancement — Quality tags are automatically prepended; model-specific negative prompts are injected (e.g., Pony-style score tags).
• cuDNN disabled — Disabled to avoid CUDNN_STATUS_NOT_INITIALIZED conflicts with the system CUDA driver; PyTorch built-in CUDA kernels are used instead.
• TunableOp — Written to disk on exit as a cache warm-up safety net.
• ControlNet models — Loaded on demand per request; not cached between requests.

License

This project is provided as-is for personal and research use.