SwarmUI StableDiffusion.cpp Backend

===========================================================================

Table of Contents

---

1. Introduction
2. Features
3. Quick Start
4. Configuration
5. Usage Tips
6. Performance and Caching
7. Troubleshooting
8. Architecture Support
9. Advanced Features
10. Technical Architecture
11. Performance Benchmarks
12. Contributing
13. License
14. Credits

Introduction

---

This extension adds a SwarmUI backend powered by stable-diffusion.cpp. It runs image generation through an external SD.cpp executable (CPU/CUDA/Vulkan) and integrates the results into SwarmUI.

Features

---

Core capabilities

• Z-Image Models - Supports Z-Image Turbo with the required Qwen LLM text encoder.
• Flux Models - Full support for FLUX.1-dev, FLUX.1-schnell, and FLUX.2-dev with automatic component management.
• SD3/SD3.5 - Multi-component architecture support (CLIP-G, CLIP-L, T5-XXL) for SD3 family models.
• SDXL/SD1.5/SD2 - Compatible with the mainstream Stable Diffusion architectures.
• Newer image models - Ideogram4, Krea2, Lens, ERNIE-Image, SeFi-Image, Mage-Flow, Boogu, LongCat-Image, PiD, MiniT2I and HiDream-O1, with the encoders and VAEs each one needs.
• Video Generation - Wan 2.1/2.2, LTX-2.3, MiniMax-H3, HunyuanVideo 1.5 and LingBot-Video, covering text-to-video, image-to-video and first/last-frame modes. LTX-2.3 and MiniMax-H3 also decode the audio track they generate.
• Image editing - Kontext, Qwen-Image-Edit (including 2509/2511), Boogu Edit, Mage-Flow-Edit, LongCat Edit and PiD, using reference-image conditioning with the preset each model expects, and accepting several reference images at once.
• IP-Adapter and ADetailer - IP-Adapter reference conditioning for SD 1.5 and SDXL, plus an optional YOLOv8-driven detail repair pass.
• GGUF Format - Load quantized GGUF models in common precisions (Q2_K, Q4_K, Q8_0).
• LoRA Support - Automatic LoRA discovery from the Models/Lora directory.
• ControlNet (experimental) - Single ControlNet per job with detection of unsupported setups.
• Live Previews - TAESD previews update frequently during generation. TODO: This needs to be tested.
• Auto-Update - Automatically downloads SD.cpp binaries from GitHub releases when enabled.

Performance optimizations

TODO: This needs work. Currently generations are slow even on repeat runs.

• Inference Caching - cache-dit/ucache/easycache integration exists but is currently not delivering the expected speedups.
• Memory Mapping - The --mmap option speeds up model loading and reduces RAM usage.
• VAE Convolution - Direct convolution (--vae-conv-direct) accelerates decoding.
• VAE Tiling - Breaks down VAE work into tiles to lower VRAM requirements.
• CPU Offloading - Move the VAE and CLIP encoders to CPU when GPU memory is constrained.
• Flash Attention - Optional attention path that saves memory at slight quality cost.

Platform support

• CUDA - NVIDIA GPUs using the installed CUDA toolkits (11.x, 12.x, or compatible 13.x drivers).
• CPU - AVX/AVX2-compatible fallback.
• Vulkan - Experimental GPU acceleration path that can work on non-NVIDIA hardware (limited Flux support).

Quick Start

---

Installation

This extension is installed like any other SwarmUI extension.

Preferred Method (Via SwarmUI)

1. Open your SwarmUI instance.
2. Navigate to Server → Extensions.
3. Find "SD.cpp Backend".
4. Click Install.
5. Restart SwarmUI when prompted (extensions require a rebuild/restart to load).

Once installed it shows up in the installed-extensions table, where you can update, disable, or uninstall it:

Manual Installation

1. Close SwarmUI.
2. Clone this repository into SwarmUI/src/Extensions/SwarmUI-SD.cpp-Backend/.
3. Restart SwarmUI.
4. Go to Server → Extensions and enable "SD.cpp Backend".

First Run

1. Open Server → Backends → SD.cpp Backend.
2. Choose your device (CPU, CUDA, or Vulkan).
3. (CUDA only) Leave CUDA version on Auto unless you know you need 11.x vs 12.x.
4. The installer downloads the SD.cpp release into dlbackend/sdcpp/{device} and then creates a run-sd-server.sh wrapper on Linux that always sets LD_LIBRARY_PATH to the binary directory before launching, so the bundled shared libraries (e.g. libstable-diffusion.so) are resolvable even on clean systems.

A green border and a running label mean the executable passed its startup probe. If it comes up errored, View Logs names the reason, and Custom Executable Path lets you point at your own sd-cli build when no prebuilt fits your system:

With a model selected, generation runs like any other backend, and previews stream into the batch strip on the right as SD.cpp works through the step count:

Add Models

Place your models in Models/Stable-Diffusion/. GGUF models go into the /diffusion_models/ folder.

Supported formats include:

• GGUF models
• SafeTensors
• CKPT/BIN

Follow the SwarmUI Supported Models documentation for more details on properly installing models into Swarm.

Configuration

---

Backend Settings

Access settings via Server → Backends → SD.cpp Backend

Device Selection:

• CPU - Universal, works on any system (slower)
• CUDA - NVIDIA GPUs (best performance)
• Vulkan - Any modern GPU (experimental for Flux)

CUDA Version (Auto-detects):

• Auto - Automatically detects your CUDA installation (recommended)
• CUDA 11.x - For older NVIDIA drivers (450+)
• CUDA 12.x - For newer NVIDIA drivers (525+)
• CUDA 13.x - Uses CUDA 12 binaries (forward compatible)

Auto-Update:

• Enabled by default
• Checks GitHub for SD.cpp updates on startup
• Downloads latest version automatically

Machine-Level Settings

Anything that describes the machine rather than an individual image lives in Server -> Backends next to the device picker, not in the parameter sidebar:

Performance Parameters

SwarmUI parameters are organized into standard Swarm groups (Sampling / Advanced Sampling / Advanced Video / Advanced Model Addons / Refine-Upscale / ControlNet), plus SD.cpp-specific groups (VRAM/Memory and Performance/Caching).

Below is the definitive list of SD.cpp-related parameters and what SD.cpp CLI arguments they emit.

The "Shown for" column matters. Parameters are gated by feature flag, so you only see what the model you have loaded can actually use. PhotoMaker does not appear for Flux, Wan options do not appear for SD 1.5, and the high-noise group only appears for Wan 2.2. Model-specific parameters appear once the model is loaded onto the backend, so switching models changes the visible set. A model whose filename defeats architecture detection falls back to showing everything rather than nothing.

These flags are namespaced (sdcpp_video, sdcpp_flux, and so on) on purpose: SwarmUI unions feature flags across every enabled backend, so an un-namespaced name could be satisfied by a different backend entirely.

Every SD.cpp parameter is marked advanced, so tick Display Advanced Options at the bottom of the parameter sidebar before you go looking for these groups:

Sampling

Swarm core group.

Advanced Sampling

Swarm core group.

SD.cpp VRAM / Memory

SD.cpp-specific group. Memory placement and VAE tiling.

SD.cpp Performance / Caching

SD.cpp-specific group. Caching strategy and load-time options.

Refine / Upscale

Swarm core group. SD.cpp's highres fix is driven from the core refiner controls here. Refiner Upscale is the switch: leave it at 1 and no highres pass runs, so you can use the ESRGAN post-upscale parameters in this group on their own. Refiner Control Percentage and Refiner Steps modify the pass but do not turn it on.

SD.cpp runs the second pass with the same model and VAE as the first, so Refiner Model, Refiner VAE, and any Refiner Method other than PostApply are rejected with an explanatory error rather than silently ignored. Refiner CFG Scale and Refiner Do Tiling have no SD.cpp equivalent and are not applied.

ControlNet

Swarm core group.

Init Image

Swarm core group.

Advanced Video

Swarm core group.

SD.cpp High-Noise Overrides

SD.cpp-specific group. Only appears for Wan 2.2. Anything left unset inherits the low-noise value.

Advanced Model Addons

Swarm core group.

CLI Coverage

Every stable-diffusion.cpp master-827 option is either emitted by a parameter above, derived from a Swarm core parameter, or exposed as a machine-level setting. The only options deliberately left out are the ones that make no sense inside SwarmUI:

--version, --list-devices, --help (used internally by the startup capability probe), --convert-name, --imat-in, --imat-out (offline quantization tooling), --metadata-all, --metadata-brief, --metadata-raw, --metadata-format, --image (CLI metadata-inspection mode), --mode (set by the backend from the model architecture), --prompt-file, --negative-prompt-file (SwarmUI passes prompts inline), --color (log colouring), --output-begin-idx (SwarmUI owns output naming), --tae (alias of --taesd, which is emitted), and --qwen2vl / --qwen2vl_vision (deprecated aliases of --llm / --llm_vision, which are emitted).

Usage Tips

---

Known Issues

---

• SD.cpp master-827 is the minimum – Device placement now goes through --backend, which upstream added in May 2026. A downloaded binary older than that is replaced automatically on startup, bypassing the once-a-day update check, because that check is per-device and a device you rarely select can otherwise sit on a months-old binary. A binary supplied through Custom Executable Path is never overwritten; that case is reported so you can rebuild it yourself.
• Linux prebuilts need glibc 2.38 – SD.cpp's official Linux binaries are built on Ubuntu 24.04, so they will not start on older distributions (Ubuntu 22.04 and similar). There is also no prebuilt CUDA binary for Linux at all. On those systems, build stable-diffusion.cpp from source and point the backend's Custom Executable Path setting at the resulting sd-cli.
• Components without an automatic download – Several newer architectures need text encoders whose upstream repositories are gated or laid out differently from their documentation. Those are not auto-downloaded; install them yourself or select them with the VAE / Text Encoder parameters. The error message names the exact file when one is missing.
• Slow generations – Initial caching is slow and repeated runs do not yet reach expected speedups across architectures. Measured on an RTX 3060 12GB with a CUDA build of master-827-97d2990: SD 1.5 at 768x768 for 40 steps took 254 seconds end to end, roughly 6.3 seconds per step. That is far off what this hardware should manage, and the cause has not been isolated yet. The default-on --diffusion-conv-direct and the per-step --preview-interval 1 are the first things worth ruling out.
• Z-Image text encoder – SD.cpp fails to load the shipped Qwen text encoder (text_encoders.llm.model.* tensors are missing), so Z-Image inference currently errors unless you manually provide a compatible encoder (GGUF version is confirmed to work).
• Previews – TAESD preview images may fail to render if the binary's capability probe does not detect --preview; expect missing preview frames when that happens.
• Img2Img/Upscaling – The backend has not been fully exercised with img2img or upscaling workflows, so their behavior remains unverified and may have undiscovered issues.

Performance and Caching

---

• TODO: The current caching/performance behavior is not acceptable. Identify why repeat generations are not speeding up as expected.
• TODO: Verify and document when cache-dit, ucache, and easycache actually apply, and what models/architectures benefit.
• TODO: Add profiling notes (CPU vs CUDA vs Vulkan), common bottlenecks, and recommended defaults.
• TODO: Add a small troubleshooting matrix for "first run slow" vs "every run slow".

Performance Optimization

For Best Speed:

1. Enable "Cache Mode" set to auto or cache-dit
2. Set "Cache Preset" to ultra or fast
3. Enable "Memory Map Models"
4. Enable "VAE Direct Convolution"
5. Use quantized GGUF models (Q4_K or Q8_0)
6. First generation builds cache (slow), subsequent generations are 5-10x faster

For Limited VRAM:

1. Enable "VAE Tiling"
2. Enable "VAE on CPU" and "CLIP on CPU"
3. Use lower quantization (Q2_K, Q4_0)
4. Reduce image dimensions
5. Close other GPU applications

Dynamic VRAM Policy (Auto Offload)

SwarmUI automatically applies SD.cpp offload flags when VRAM is tight. This system is always on and uses real model file sizes, GPU free VRAM, and generation parameters (resolution + batch count) to decide which flags are needed. It does not clear VRAM between generations, so models stay resident and repeat runs stay fast.

How it works:

• Computes an estimated VRAM footprint from model sizes + runtime overhead + resolution
• Compares that to free VRAM with a safety margin
• Gradually escalates offload flags only if required

Escalation order (more aggressive as needed):

1. --vae-tiling
2. text encoders to CPU
3. VAE to CPU
4. --offload-to-cpu

Steps 2 and 3 are emitted as a single --backend te=cpu,vae=cpu assignment. SD.cpp master-827 deprecated the old per-module --clip-on-cpu / --vae-on-cpu / --control-net-cpu flags, and this backend no longer emits them at all, which is why master-827 is the minimum version. The backend probes --help on startup and refuses to start with a clear message if the binary predates --backend, rather than starting and silently skipping offload.

Notes:

• User-set flags are respected if they are more aggressive than the auto-policy.
• Very low VRAM GPUs (<6 GB) will automatically enable all offload flags.
• Setting Backend Assignment by hand overrides this entire ladder.
• Choosing the Auto (SD.cpp --auto-fit) VRAM policy skips this ladder and lets SD.cpp place modules itself. The two cannot both run, because they make the same decisions from the same inputs.

LoRA Usage

• TODO: Test and verify LoRA functionality with various models.

Important:

• Place LoRA files in Models/Lora/
• Backend automatically detects LoRA directory

Video Generation (Wan Models)

Video-specific parameters:

• Video Frames: Number of frames to generate
• Video FPS: Frames per second
• Flow Shift: Flow control (default 3.0 for Wan)
• Wan 2.2: Supports dual-model system with high-noise diffusion model

Troubleshooting

---

No Preview Images

Problem: Live previews not showing during generation

• TODO: This is still a work in progress and needs testing.

Solution:

• Backend automatically enables TAESD previews
• Previews require an SD.cpp build with --preview; the backend probes --help on startup and logs the result
• MiniMax-H3 needs SD.cpp master-827 or newer; older builds refuse --taesd on H3 and log a warning instead
• Check logs to verify --preview tae is being used
• Previews update every step or every 500ms

Out of Memory

Problem: CUDA out of memory or allocation failed

Solution:

1. Enable "VAE Tiling"
2. Enable "VAE on CPU"
3. Use lower quantization (Q2_K instead of Q8_0)
4. Reduce image dimensions
5. Close other GPU applications

Architecture Support

---

Generated from the architecture table in Models/SDcppArchitectures.cs, so it stays in step with the code.

Status legend: Arg-verified means the command line this extension builds for that architecture was checked against stable-diffusion.cpp master-827-97d2990 - every flag exists, components resolve, and the routing matches that model's upstream docs. It does not mean a generation was run on that architecture.

Architectures whose extra components have no verified download are resolved from what you already have installed (matched by filename) or picked explicitly with the VAE / Text Encoder parameters. If one is missing, generation stops with an error naming the exact file and the folder to put it in, rather than failing partway through. The Flux, Flux.2, Wan and CLIP-Vision components do download automatically.

Advanced Features

---

Img2Img and Inpainting

• TODO: Test and verify img2img and inpainting functionality.

• Init Image: Automatic img2img support

• Init Image Creativity: Strength parameter (0.0-1.0)

• Mask Image: Inpainting with mask support

ControlNet (Experimental)

• SD.cpp supports single ControlNet
• Backend will warn if multiple ControlNets used
• ControlNet model + control image required
• Control strength parameter (0.0-2.0)

Batch Generation

• TODO: Test and verify batch generation functionality.

• Batch count parameter (generates multiple images)

• SD.cpp --batch-count flag

• All images saved and returned

ESRGAN Upscaling

• TODO: Test and verify ESRGAN upscaling functionality.

• Post-processing upscaler

• Supports RealESRGAN models

• Multiple upscale passes supported

• Place upscale models in Models/upscale_model/

Technical Architecture

---

Extension Structure

SwarmUI-SD.cpp-Backend/
├── SDcppExtension.cs              # Main extension entry point
├── SwarmBackends/
│   └── SDcppBackend.cs            # Backend implementation (~400 lines, refactored)
├── Models/
│   ├── SDcppModelManager.cs       # Model detection, validation, downloads
│   └── SDcppParameterBuilder.cs   # Parameter conversion to SD.cpp CLI format
├── Utils/
│   ├── GGUFConverter.cs           # GGUF conversion helpers
│   ├── SDcppDownloadManager.cs    # Auto-download SD.cpp binaries
│   ├── SDcppProcessManager.cs     # Process execution and output capture
│   └── SDcppVramPolicy.cs         # Dynamic VRAM offload policy
└── WebAPI/
    └── SDcppAPI.cs                # Additional API endpoints

File Format Support

Model Formats:

• .gguf - Native SD.cpp format (Q2_K, Q4_K, Q8_0 quantization)
• .safetensors - Standard format (recommended)
• .ckpt, .pth - PyTorch checkpoint formats

Image Formats: Input (SD.cpp CLI):

• .png, .jpg/.jpeg, .bmp

Output (SD.cpp CLI):

• .png by default
• .jpg/.jpeg/.jpe when the output path uses a JPEG extension

SwarmUI backend:

• SD.cpp output is requested as PNG; SwarmUI can convert images to other formats if needed.

Performance Benchmarks

---

One measured data point so far. Treat it as a baseline to beat, not as a target.

Flags in play for that run: --diffusion-fa --diffusion-conv-direct --mmap --vae-conv-direct --cache-mode ucache --cache-option reset=1 --preview proj --preview-interval 1.

• TODO: Add CPU and Vulkan numbers for the same settings.
• TODO: Measure with --diffusion-conv-direct and per-step previews disabled, to find where the time actually goes.
• TODO: Include "first run" vs "repeat run" measurements and note when (if ever) caching improves throughput.

Contributing

---

Contributions welcome! Focus areas:

• Performance profiling and optimization Image gen is very slow currently.
• Better error messages and user guidance
• UI/UX improvements
• Swarm parameter fixes

License

---

MIT License

Credits

---

• stable-diffusion.cpp by leejet
• SwarmUI by mcmonkey

---

Last Updated: August 2026 Extension Version: 0.1.6 Pinned SD.cpp Version: master-827-97d2990