Project Status: Active Development

Thank you for the incredible support and over 30,000 downloads!

ollamadiffuser is in active development. v2.5.0: 106 models — 44 native on Apple Silicon through MLX (38 image models, 6 LTX-2 video packs) and 62 through diffusers — with one server for every model, loaded on demand, one-command image, edit and video jobs for scripts and agents, on this machine or another one's GPU, Qwen-Image-2.1 editing and its official 8-step Turbo on a Mac, video on Mac and NVIDIA, LoRAs on both backends, and a compile-free install with a one-line curl | sh installer. See What's New below. Part of the LocalKinAI ecosystem.

🆕 What's New

v2.5.0 — one server for every model

• ollamadiffuser serve loads the model each request names. It goes in model: a field in /api/generate's JSON, or a form field for img2img, inpaint, ControlNet and video.
  • --max-loaded 2 keeps two models in memory, so going back and forth between a drawing model and an editing model costs no reload. Jobs still run one at a time.
  • --keep-alive 10m gives an idle model's memory back.
  • run MODEL still serves that model only.
  • See REST API.
• The memory check goes by measured peaks. Every generation records what its model took, and the check uses that number instead of the registry's estimate. Boogu Image Turbo measured 32.1 GB, against 10 GB listed.
• Upgrading: restart servers started before the upgrade. Full notes in CHANGELOG.md.

v2.4.0 — jobs on another machine's GPU

• --server 192.168.0.21:8010 on generate, edit and video, or OLLAMADIFFUSER_SERVER set once (MCP reads it too), sends the job to the OllamaDiffuser server on another machine. The pictures go up, the result comes back here, and nothing loads on this machine. ps --host 192.168.0.21 lists what runs there. See Using OllamaDiffuser from agents.
• Fixed: processes sharing ~/.ollamadiffuser/config.json wrote over each other's changes. Each save now writes only its own changes, under a lock, in one rename. Restart servers started before the upgrade. Full notes in CHANGELOG.md.

v2.3.0 — one command for agents

Upgrade with pip install -U ollamadiffuser; full notes in CHANGELOG.md. A failed generation is now an error from the API rather than a 200 with a picture of the error, and on a Mac a model that won't fit in free memory isn't loaded unless forced.

• ollamadiffuser generate / edit / video make a picture, an edit or a clip in one command, and print the file (--json for scripts and agents). The job goes to a server that already holds the model, or loads it for the job. See Using OllamaDiffuser from agents.
• ps --json, list --json, stop --port N / --all; MCP gains edit_image, generate_video and list_servers; a Claude Code skill in integrations/claude-code/.
• On a Mac, a model that won't fit in free memory isn't loaded (--force to insist), and a failed generation is now an error rather than a picture of one.

v2.2.0 — Qwen-Image-2.1 edits and its Turbo on a Mac

The first PyPI release since 2.1.0. Upgrade with pip install -U ollamadiffuser, then ollamadiffuser enable mlx on a Mac (mflux 0.22); full notes in CHANGELOG.md.

• Edit with Qwen-Image-2.1 on Apple Silicon. qwen-image-2.1-edit-mlx takes up to ten reference pictures, a mask, an edit strength and prompt rewriting, from the same weights as qwen-image-2.1-mlx. Its ControlNet (qwen-image-2.1-controlnet-mlx: canny, depth, pose and five more, plus inpainting) and Qwen-Image-Edit-2511 (qwen-image-edit-2511-mlx) run natively too. Needs mflux 0.22, now the minimum.
• Qwen-Image-2.1 in 6 steps. Load Viggle's turbo LoRA and pass scheduler=viggle_turbo — see LoRAs on Apple Silicon.
• New: ming-image-design-mlx (UI, posters and infographics with transparent backgrounds, MIT); Qwen's own 8-step Qwen-Image-2.1-Turbo on both platforms — qwen-image-2.1-turbo-mlx (sampled on the schedule the checkpoint ships with, which mflux has no entry for; 32 s an image at 1024² on an M3 Ultra) and qwen-image-2.1-turbo through diffusers 0.41 — and qwen-image-2.1 on NVIDIA.
• Fixed: qwen-image-edit-mlx redrew the picture instead of editing it — it was built as the text-to-image class. The MCP server didn't start with mcp 2.x, which ollamadiffuser enable mcp installs. MLX models hold MLX's buffer cache to 2 GB (OLLAMADIFFUSER_MLX_CACHE_LIMIT_GB): by default MLX lets it grow to the size of the machine, and one 1024² Qwen-Image-2.1-Turbo image took a 96 GB Mac from 28 GB free to under 5 GB, though it needed 15.5 GB.
• Better: pull links files another installed entry already downloaded from the same repo, so an editor next to its generator costs no second download. ollamadiffuser ps lists every running server, not just the last one started. MCP's generate_image also returns the saved file's path as text, and uses a running server that already holds the model instead of loading a second copy.

v2.1.1 — ComfyUI nodes

Three nodes that let a ComfyUI workflow use OllamaDiffuser's models — Qwen-Image-2.1, FLUX.2 klein, Kontext, Boogu, LTX-2 on MLX — as ordinary IMAGE / VIDEO steps, through the HTTP API, with no second copy of the weights. See ComfyUI Nodes.

v2.1.0 — video and LoRAs on every platform, 40 more models

The first PyPI release since 2.0.17, so it also brings 2.0.18–2.0.26 below to pip. Full notes in CHANGELOG.md.

• 🎬 Video on NVIDIA. A new diffusers-video path runs wan2.1-t2v-1.3b, wan2.2-ti2v-5b and ltx-2.3-distilled (with its soundtrack) behind the same POST /api/generate/video as LTX-2 on Apple Silicon. On the Mac, LTX-2 can now follow a reference video — a pose skeleton, depth or edges — for motion no prompt can describe, and runs offline.
• 🎛️ LoRAs on Apple Silicon. ollamadiffuser lora load works on -mlx models: the model is rebuilt with the LoRA baked in, and LoRAs stack — a speed LoRA and a style LoRA together. FLUX.1, FLUX.2 klein, Z-Image, Qwen-Image, Krea 2, ERNIE. LTX video takes style and motion LoRAs too.
• 🖼️ New models. qwen-image-2.1-mlx (108 s per 1024² image on an M3 Ultra, faster than ComfyUI; research licence), qwen-image-edit-2511, the undistilled bases z-image, flux.2-klein-base-4b, ernie-image, krea-2-raw, flux.1-krea-dev, the anime fine-tune z-anime, seedvr2-7b-mlx, z-image-turbo-controlnet-mlx (one ControlNet for canny, depth, pose, HED, MLSD), realvisxl-v5.
• 🖥️ Not just for Macs. Nine models that had been Mac-only now have diffusers entries for NVIDIA: z-image, flux.2-klein-base-4b, flux.2-klein-9b, ernie-image, ernie-image-turbo, krea-2-turbo, krea-2-raw, qwen-image, flux.1-krea-dev.
• ♻️ Smaller downloads. pull hard-links weights already in the Hugging Face cache (from ComfyUI, mflux or hf download) instead of downloading them again, and entries stop fetching files their loader never opens — krea-2-turbo-mlx 62 → 36 GB, ernie-image-turbo-mlx 32 → 24 GB, Z-Anime 17.5 GB of a 202 GB repo.
• 🔌 New endpoints. POST /api/face/compare (is it still the same person?), several reference pictures on /api/generate/img2img, control_type on /api/generate/controlnet.
• Upgrading: pip install -U ollamadiffuser. Two new dependencies, av and ftfy, are prebuilt wheels. The MLX backend needs mflux 0.20.0 (ollamadiffuser enable mlx). Licence correction: flux.2-klein-9b-mlx is FLUX Non-Commercial, not Apache 2.0.

v2.0.26 — the registry pulls what the loader loads

qwen-image-mlx / qwen-image-edit-mlx were pulling 58 GB of a checkpoint mflux never loads (now Qwen-Image-2512 / Qwen-Image-Edit-2509), and seedvr2-3b-mlx fetched a 3.4 GB fp8 file it never opened. ollamadiffuser enable gguf now builds with CUDA on NVIDIA machines instead of silently producing a CPU build. New: CONTRIBUTING.md — adding a model is a data-only PR.

v2.0.22 — video, natively: LTX-2 on Apple Silicon

Six new entries (ltx-2.3-mlx-{q4,q8,bf16}, ltx-2.5-mlx-{q4,q8,bf16}) drive ltx-2-mlx for text-to-video with audio, image-to-video and audio-to-video — plus a POST /api/generate/video endpoint and engine.generate_video(). See the video section.

v2.0.21 — eight more MLX families: Krea 2, Boogu, ERNIE-Image, Lens, Ideogram 4, FIBO, FIBO-Edit, SeedVR2

mflux grew a lot of model families after our May line, and the registry had not caught up. Eight new entries — 22 MLX entries total — covering photographic turbo models (Boogu 4-step with bilingual EN/ZH text, ERNIE-Image, Krea 2), a 4-step Microsoft model with a 20B text encoder (Lens), typography (Ideogram 4), JSON-prompted generation and editing (FIBO, FIBO-Edit) and the best open upscaler (SeedVR2). Each is a class and an alias: mflux resolves the config from the alias itself, so adding a family is data plus one line of routing.

v2.0.20 — One-line curl | sh installer (no system Python needed)

curl -fsSL .../install.sh | sh now installs a fully isolated OllamaDiffuser via uv: a standalone Python + the compile-free core under ~/.ollamadiffuser, with a launcher on your PATH. It doesn't touch your system Python, needs no compiler, and auto-enables MLX on Apple Silicon — the Ollama-style zero-friction install.

v2.0.19 — Model registry is now data (models.yaml)

The 59 built-in models moved from a 1500-line Python dict into a bundled models.yaml. Adding a model is now a pure-data PR — no code. Migration is zero-loss (snapshot-pinned in tests).

v2.0.18 — Compile-free default install + enable command

The default pip install ollamadiffuser is now compile-free (prebuilt wheels only — no CMake, no CUDA toolchain). Optional backends are opt-in by name via a new command: ollamadiffuser enable mlx | gguf | mcp — no shell-quoted [extras] to get wrong, and enable gguf sets the right CMAKE_ARGS (Metal on Mac) for you. The curl | sh installer now defaults to the lean core and auto-enables MLX on Apple Silicon.

v2.0.17 — MLX Phase 2.5: FLUX.1 family completion

MLXStrategy now covers the full FLUX.1 family on Apple Silicon: flux1-fill (inpaint/outpaint), flux1-redux (image variation), flux1-depth (depth-conditioned), flux1-controlnet (canny + upscaler). Five new registry entries — 14 MLX entries total (see the MLX section below).

v2.0.15–v2.0.16 — MLX Backend (Phases 1 + 2)

New MLXStrategy routes FLUX.1 / FLUX.2 / Z-Image / Qwen-Image / Kontext through mflux for native Apple Silicon inference. Typically 2-3× faster than the PyTorch + MPS path. Install with pip install 'ollamadiffuser[mlx]'. Tracks #7.

v2.0.14 — Diffusers Pipeline Additions

• flux.1-kontext-dev (PyTorch) — 12B instruction-based image editing. Pass an input image + edit prompt; the model rewrites the image.
• chroma1-hd — 8.9B Apache-2.0 base T2I (FLUX-schnell derivative). Rare commercial-friendly license at this quality tier.

v2.0.13 — Bug Fixes + Discussions

• Fixed ollamadiffuser recommend crash on CUDA hosts (PyTorch attribute typo).
• GitHub Discussions enabled: https://github.com/LocalKinAI/ollamadiffuser/discussions
• 18 broken org-name URLs fixed across PyPI metadata, README, and guides.

See CHANGELOG.md for the full history (back to v1.0.0, May 2025).

OllamaDiffuser 🎨

Local AI Image Generation with OllamaDiffuser

OllamaDiffuser simplifies local deployment of Stable Diffusion, FLUX, CogView4, Kolors, SANA, PixArt-Sigma, and 40+ other AI image generation models. An intuitive local SD tool inspired by Ollama's simplicity - perfect for local diffuser workflows with CLI, web UI, and LoRA support.

🌐 Website: ollamadiffuser.com | 📦 PyPI: pypi.org/project/ollamadiffuser

Upgrading from v1.x? v2.0 is a major rewrite requiring Python 3.10+. Run pip install --upgrade ollamadiffuser (then ollamadiffuser enable gguf/mlx if you need those backends) and see the Migration Guide below.

---

🚀 Quick Start

Zero-dependency install (recommended — no Python setup needed)

curl -fsSL https://raw.githubusercontent.com/LocalKinAI/ollamadiffuser/main/install.sh | sh

This installs an isolated Python + OllamaDiffuser under ~/.ollamadiffuser (via uv — a single static binary) and drops an ollamadiffuser launcher on your PATH. It never touches your system Python, needs no pip/venv/compiler, and on Apple Silicon auto-enables the MLX backend. Uninstall is rm -rf ~/.ollamadiffuser.

Already have a Python env?

The default install is compile-free — prebuilt wheels only, no CMake and no CUDA toolchain. Optional backends are opt-in by name, so you never fight bracket-quoting in your shell.

pip install ollamadiffuser
ollamadiffuser recommend   # Find which models fit your hardware

Mac / Apple Silicon (recommended — typically 2-3× faster):

pip install ollamadiffuser
ollamadiffuser enable mlx  # native MLX backend, still compile-free

OpenClaw / Agent users:

pip install ollamadiffuser
ollamadiffuser enable mcp  # Model Context Protocol server deps
ollamadiffuser mcp         # start the server

Low-VRAM / GGUF (advanced — compiles a native extension):

pip install ollamadiffuser
ollamadiffuser enable gguf                 # sets Metal/CUDA build flags for you
ollamadiffuser pull flux.1-dev-gguf-q4ks   # only 6GB VRAM needed
ollamadiffuser run  flux.1-dev-gguf-q4ks

Why enable instead of pip install "ollamadiffuser[gguf]"? The GGUF backend compiles a native library (stable-diffusion-cpp-python). ollamadiffuser enable gguf runs that build with the right CMAKE_ARGS for your platform (Metal on Mac) so you never have to remember them — and there are no shell-quoted [extras] to get wrong. The bracketed extras ([gguf], [mlx], [mcp], [full]) still work if you prefer them.

Most models work without any token -- just install and go. See Hugging Face Authentication when you want gated models like FLUX.1-dev or SD 3.5.

---

✨ Features

• 🏗️ Strategy Architecture: Clean per-model strategy pattern (SD1.5, SDXL, FLUX, SD3, ControlNet, Video, LTX-2 video (MLX), HiDream, GGUF, MLX, Generic)
• 🌐 60+ Models: FLUX.1/2, SD 3.5, SDXL Lightning, CogView4, Kolors, SANA, PixArt-Sigma, Z-Image, Qwen-Image, Chroma1, and more
• 🔌 Generic Pipeline: Add new diffusers models via registry config alone -- no code changes needed. Built-in models live in models.yaml (data, not code) — contributing a model is a pure-data PR against that file.
• 🖼️ img2img & Inpainting: Image-to-image and inpainting support across SD1.5, SDXL, and the API/Web UI
• ⚡ Async API: Non-blocking FastAPI server using asyncio.to_thread for GPU operations
• 🎲 Random Seeds: Reproducible generation with explicit seeds, random by default
• 🎛️ ControlNet Support: Precise image generation control with 10+ control types (PyTorch + MLX)
• 🔄 LoRA Integration: Dynamic LoRA loading and management
• 🔌 MCP & OpenClaw: Model Context Protocol server for AI assistant integration (OpenClaw, Claude Code, Cursor)
• 🧩 ComfyUI nodes: draw, edit and film with OllamaDiffuser's models from inside a ComfyUI graph (integrations/comfyui/)
• 🍎 Apple Silicon, two paths:
  • MLX backend via mflux — 38 native MLX entries (FLUX.1 family, FLUX.1 Krea, FLUX.2 Klein, Z-Image, Qwen-Image, Qwen-Image-2.1 with editing and a ControlNet, Kontext, Fill, Redux, Depth, ControlNet, and since v2.0.21 Krea 2, Boogu, ERNIE-Image, Lens, Ideogram 4, FIBO, FIBO-Edit, SeedVR2, Ming-Image). Typically 2-3× faster than the PyTorch + MPS path on M-series.
  • PyTorch + MPS — full diffusers pipeline support with per-model dtype handling (float16/bfloat16, NaN sanitization), GGUF Metal acceleration, and ollamadiffuser recommend for hardware-aware model suggestions.
• 📦 Smart Downloads: ollamadiffuser pull downloads only diffusers pipeline files — skips root-level checkpoints, ONNX/Flax exports, and safety_checker. Saves 10–200 GB per model.
• ♻️ Reuses what's already downloaded: if a model is already in the Hugging Face cache (from ComfyUI, mflux, hf download…), ollamadiffuser pull hard-links it instead of downloading — seconds instead of tens of GB, and no second copy on disk.
• 📦 GGUF Support: Memory-efficient quantized models (3GB VRAM minimum!) with CUDA and Metal acceleration
• 🌐 Multiple Interfaces: CLI, Python API, Web UI, and REST API
• 📦 Model Management: Easy installation and switching between models
• ⚡ Performance Optimized: Memory-efficient with GPU acceleration
• 🧪 Test Suite: 243 tests across settings, registry, engine, API, MPS, MLX, LTX-2 video, and MCP

Option 1: Install from PyPI (Recommended)

# Install from PyPI
pip install ollamadiffuser

# Pull and run a model
ollamadiffuser pull flux.1-schnell
ollamadiffuser run flux.1-schnell

# Generate via API (seed is optional for reproducibility)
curl -X POST http://localhost:8000/api/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt": "A beautiful sunset", "seed": 12345}' \
  --output image.png

🔄 Update to Latest Version

Always use the latest version for the newest features and bug fixes:

# Update to latest version
pip uninstall ollamadiffuser
pip install --no-cache-dir ollamadiffuser

This ensures you get:

• 🐛 Latest bug fixes
• ✨ New features and improvements
• 🚀 Performance optimizations
• 🔒 Security updates

GGUF Quick Start (Low VRAM)

# Enable the GGUF backend (compiles once; sets the right build flags for you)
ollamadiffuser enable gguf

# Download a memory-efficient GGUF model (3GB+ VRAM)
ollamadiffuser pull flux.1-dev-gguf-q4ks

# Generate with reduced memory usage
ollamadiffuser run flux.1-dev-gguf-q4ks

Apple Silicon Quick Start (Mac Mini / MacBook)

# See which models fit your Mac
ollamadiffuser recommend

# Fast single-step model (<6GB)
ollamadiffuser pull sdxl-turbo
ollamadiffuser run sdxl-turbo

# Native MLX path — typically 2-3x faster than PyTorch+MPS (auto-enabled
# by the curl|sh installer; run this if you installed via pip)
ollamadiffuser enable mlx
ollamadiffuser pull flux.1-schnell-mlx
ollamadiffuser run flux.1-schnell-mlx

# GGUF with Metal acceleration (6GB, great quality)
ollamadiffuser enable gguf   # sets CMAKE_ARGS=-DSD_METAL=ON under the hood
ollamadiffuser pull flux.1-dev-gguf-q4ks
ollamadiffuser run flux.1-dev-gguf-q4ks

Option 2: Development Installation

# Clone the repository
git clone https://github.com/LocalKinAI/ollamadiffuser.git
cd ollamadiffuser

# Install dependencies
pip install -e .

Basic Usage

# Check version
ollamadiffuser -V

# Install a model
ollamadiffuser pull stable-diffusion-1.5

# Run the model (loads and starts API server)
ollamadiffuser run stable-diffusion-1.5

# Generate an image via API
curl -X POST http://localhost:8000/api/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt": "a beautiful sunset over mountains"}' \
  --output image.png

# Start web interface
ollamadiffuser --mode ui

open http://localhost:8001

ControlNet Quick Start

# Install ControlNet model
ollamadiffuser pull controlnet-canny-sd15

# Run ControlNet model (loads and starts API server)
ollamadiffuser run controlnet-canny-sd15

# Generate with control image
curl -X POST http://localhost:8000/api/generate/controlnet \
  -F "prompt=a beautiful landscape" \
  -F "control_image=@your_image.jpg"

---

🔑 Hugging Face Authentication

Do you need a Hugging Face token? It depends on which models you want to use!

Models that DON'T require a token -- ready to use right away:

• FLUX.1-schnell, Stable Diffusion 1.5, DreamShaper, PixArt-Sigma, SANA 1.5, most ControlNet models

Models that DO require a token:

• FLUX.1-dev, Stable Diffusion 3.5, some premium LoRAs

Setup (only needed for gated models):

# 1. Create account at https://huggingface.co and generate an access token
# 2. Accept license on the model page (e.g. FLUX.1-dev, SD 3.5)
# 3. Set your token
export HF_TOKEN=your_token_here

# 4. Now you can access gated models
ollamadiffuser pull flux.1-dev
ollamadiffuser pull stable-diffusion-3.5-medium

Tips: Use "read" permissions for the token. Your token stays local -- never shared with OllamaDiffuser servers. Add export HF_TOKEN=... to ~/.bashrc or ~/.zshrc to make it permanent.

---

🎯 Supported Models

Choose from 40+ models spanning every major architecture:

Core Models

Next-Generation Models

Fast / Turbo Models

Community Fine-Tunes

FLUX Pipeline Variants

Apache-2.0 Commercial-Friendly

💾 GGUF Models - Reduced Memory Requirements

GGUF quantized models enable running FLUX.1-dev on budget hardware:

📖 Complete GGUF Guide - Hardware recommendations, installation, and optimization tips

🍎 MLX Models — Apple Silicon native

MLX entries run through mflux on Apple Silicon (M1/M2/M3/M4). On M-series hardware they are typically 2-3× faster than the same model on the PyTorch + MPS path. Install with pip install 'ollamadiffuser[mlx]'.

Text-to-image:

Image editing / control:

🎬 Video — LTX-2 on Apple Silicon

Video entries (model_type: ltx-video-mlx) run LTX-2 through ltx-2-mlx, a pure-MLX port: text-to-video with 48 kHz stereo audio, image-to-video, audio-to-video, and LTX-2.5's predicted durations. This is the one strategy that shells out rather than importing — ltx-2-mlx is a three-package monorepo installed with uv sync that manages its own weight packs, and its CLI is the surface its author supports.

Experimental. Both the runtime and the weight packs (dgrauet/ltx-2.*-mlx) are one author's community port, not Lightricks' upstream release, and they are young — expect the CLI and pack layout to move. If a pack is renamed or withdrawn these entries stop pulling; please open an issue if that happens.

# One-time: the runtime it drives
git clone https://github.com/dgrauet/ltx-2-mlx && cd ltx-2-mlx && uv sync --all-extras
# then put .venv/bin on PATH, or export LTX2MLX_BIN=/path/to/ltx-2-mlx

ollamadiffuser pull ltx-2.3-mlx-q8
ollamadiffuser run ltx-2.3-mlx-q8

# Text to video (4s at 24fps). Frame counts are 8k+1; `seconds` rounds for you.
curl -X POST http://localhost:8000/api/generate/video \
  -F prompt="a courtyard in the rain, slow pan" -F seconds=4 -o clip.mp4

# Image to video — animate a still
curl -X POST http://localhost:8000/api/generate/video \
  -F prompt="she turns and smiles" -F [email protected] -F seconds=4 -o clip.mp4

# Audio to video — a voice drives the face
curl -X POST http://localhost:8000/api/generate/video \
  -F prompt="a woman speaking to camera" -F [email protected] -o clip.mp4

Video models refuse the image endpoints rather than returning one frame of the clip, and /api/generate/video answers 400 (not 500) for the things a caller can fix: no model loaded, an image model loaded, ltx-2-mlx not installed, a frame count off the grid.

🎬 Video on NVIDIA — Wan and LTX-2 through diffusers

Off Apple Silicon, video goes through diffusers' own pipelines (model_type: diffusers-video). Same endpoint and the same mp4 back: POST /api/generate/video with a prompt, and optionally image (first frame), frames or seconds, width, height, steps, cfg_scale, seed.

LTX-2 writes its soundtrack into the mp4. The LTX-2 diffusers entries refuse the Apple GPU — diffusers' LTX-2 uses float64, which Metal lacks — and point to the ltx-*-mlx entries instead.

Hardware fit at a glance:

• Mac Mini M4 16 GB can run anything marked ✅ above (Q4 FLUX.1-schnell, FLUX.2 Klein 4B, Z-Image-Turbo).
• Mac Pro M1 32 GB / M2 Pro 32 GB+ can run all entries except the 20B Qwen-Image at the larger resolutions.

Quick start:

pip install 'ollamadiffuser[mlx]'
ollamadiffuser pull z-image-turbo-mlx    # smallest Apache-2.0 option
ollamadiffuser run z-image-turbo-mlx

---

🎛️ ControlNet Features

⚡ Lazy Loading Architecture

New in v1.1.0: ControlNet preprocessors use intelligent lazy loading:

• Instant Startup: ollamadiffuser --help runs immediately without downloading models
• On-Demand Loading: Preprocessors initialize only when actually needed
• Automatic Initialization: Seamless loading when uploading control images
• User Control: Manual initialization available for pre-loading

Available Control Types

• Canny Edge Detection: Structural control with edge maps
• Depth Estimation: 3D structure control with depth maps
• OpenPose: Human pose and body position control
• Scribble/Sketch: Artistic control with hand-drawn inputs
• Advanced Types: HED, MLSD, Normal, Lineart, Anime Lineart, Content Shuffle

ControlNet Models

# SD 1.5 ControlNet Models
ollamadiffuser pull controlnet-canny-sd15
ollamadiffuser pull controlnet-depth-sd15
ollamadiffuser pull controlnet-openpose-sd15
ollamadiffuser pull controlnet-scribble-sd15

# SDXL ControlNet Models
ollamadiffuser pull controlnet-canny-sdxl
ollamadiffuser pull controlnet-depth-sdxl

🔄 LoRA Support

Dynamic LoRA Management

# Download LoRA from Hugging Face
ollamadiffuser lora pull "openfree/flux-chatgpt-ghibli-lora"

# Load LoRA with custom strength
ollamadiffuser lora load ghibli --scale 1.2

# Unload LoRA
ollamadiffuser lora unload

LoRAs on Apple Silicon (MLX models)

The same commands work on -mlx models. mflux takes a LoRA when the model is built, so loading or unloading one rebuilds the model with it baked in, and costs nothing per step afterwards. Measured on an M3 Ultra with FLUX.2 klein 4B: 45 s to load the LoRA the first time, most of it the 325 MB download; 0.3 s to unload it. LoRAs stack: load a speed LoRA and a style LoRA and both apply. The rebuild releases the old weights before it reads the new ones, and refuses — with a sentence saying how much it needs — when the Mac has no room for one more copy.

ollamadiffuser lora pull Limbicnation/pixel-art-lora -w pytorch_lora_weights.safetensors -a pixel
ollamadiffuser lora load pixel --scale 1.0

Families that take LoRAs: FLUX.1 (and Kontext), FLUX.2 klein (and its edit variant), Z-Image, Qwen-Image, Qwen-Image-2.1 (generation, edit and ControlNet), Krea 2, ERNIE-Image. Boogu, Lens, Ideogram 4, FIBO, Ming-Image and SeedVR2 don't in mflux yet, and say so.

Qwen-Image-2.1 in 6 steps — Viggle's turbo LoRA with mflux's matching scheduler, for generation and editing alike. Measured on an M3 Ultra at 1024×1024: about 27 s per image (37 s for the first) against 110 s for 25 steps without it, with the sign's Chinese text still right. The Qwen-Image-2.1 entries keep LoRAs as live adapters (bake_lora: false): baked into the 8-bit weights, this one lost more than half its effect.

ollamadiffuser lora pull Viggle/Qwen-Image-2.1-viggle-turbo \
  -w Qwen-Image-2.1-viggle-turbo-v0.3-6step-lora-r256.safetensors -a qwen21-turbo
ollamadiffuser lora load qwen21-turbo
# then generate with 6 steps and the matching scheduler:
curl -s localhost:8000/api/generate -H 'Content-Type: application/json' \
  -d '{"prompt": "a cabin by the sea", "steps": 6, "scheduler": "viggle_turbo"}' -o cabin.png

On LTX video, pass lora= (a local .safetensors or a hub repo id) and lora_strength= to /api/generate/video for a style or motion LoRA; with a control video, lora is the IC-LoRA.

Web UI LoRA Integration

• Easy Download: Enter Hugging Face repository ID
• Strength Control: Adjust LoRA influence with sliders
• Real-time Loading: Load/unload LoRAs without restarting
• Alias Support: Create custom names for your LoRAs

🌐 Multiple Interfaces

Command Line Interface

# One picture, one edit, one clip: each prints the file it wrote
ollamadiffuser generate "a red fox in snow" -m z-image-turbo-mlx -o fox.png
ollamadiffuser edit "make it a snowy night" -i cabin.png -m qwen-image-2.1-edit-mlx
ollamadiffuser video "waves at dusk" -m ltx-2.3-mlx-q8 -s 4

# Pull and run a model as a server
ollamadiffuser pull stable-diffusion-1.5
ollamadiffuser run stable-diffusion-1.5

# What is running, and stopping it
ollamadiffuser ps                     # every server on this machine and its model
ollamadiffuser stop --port 8010       # or --all

# Model registry management
ollamadiffuser registry list
ollamadiffuser registry list --installed-only
ollamadiffuser registry check-gguf

# Configuration management
ollamadiffuser config                                    # show all config
ollamadiffuser config set models_dir /mnt/ssd/models     # custom model path
ollamadiffuser config set server.port 9000               # change server port

# In another terminal, generate images via API
curl -X POST http://localhost:8000/api/generate \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "a futuristic cityscape",
    "negative_prompt": "blurry, low quality",
    "num_inference_steps": 30,
    "guidance_scale": 7.5,
    "width": 1024,
    "height": 1024
  }' \
  --output image.png

Web UI

# Start web interface
ollamadiffuser --mode ui
Open http://localhost:8001

Features:

• Responsive Design: Works on desktop and mobile
• Real-time Status: Model and LoRA loading indicators
• ControlNet Integration: File upload with preprocessing
• Parameter Controls: Intuitive sliders and inputs

REST API

# Start a server. It loads the model each request names, keeps up to two in
# memory, and with --keep-alive gives a model's memory back after 10 idle minutes.
ollamadiffuser serve --max-loaded 2 --keep-alive 10m

# Text-to-image
curl -X POST http://localhost:8000/api/generate \
  -H "Content-Type: application/json" \
  -d '{"model": "stable-diffusion-1.5", "prompt": "a beautiful landscape", "width": 1024, "height": 1024, "seed": 42}' \
  --output landscape.png

# Image-to-image (another model: the server loads it beside the first)
curl -X POST http://localhost:8000/api/generate/img2img \
  -F "model=flux.2-klein-4b-edit-mlx" \
  -F "prompt=oil painting style" \
  -F "[email protected]" \
  --output result.png

# Inpainting
curl -X POST http://localhost:8000/api/generate/inpaint \
  -F "model=stable-diffusion-1.5" \
  -F "prompt=a red car" \
  -F "[email protected]" \
  -F "[email protected]" \
  --output inpainted.png

# API docs: http://localhost:8000/docs

With --max-loaded N (default 1), a request for a model already in memory starts at once. One for another model loads it, and the least recently used model gives way when there are N already, or when memory runs short. Jobs still run one at a time: on one GPU, two at once finish no sooner and need the memory twice. A model that could not fit even with every other model unloaded is refused before anything is unloaded.

ollamadiffuser run MODEL starts a server for that model only, and answers a request for another with 409. Both kinds take --keep-alive. /api/models/running says which kind a server is, lists the models it holds, and says when each will be let go.

MCP Server (AI Assistant Integration)

OllamaDiffuser includes a Model Context Protocol server for integration with AI assistants like OpenClaw, Claude Code, and Cursor.

# Install MCP support
pip install "ollamadiffuser[mcp]"

# Start MCP server (stdio transport)
ollamadiffuser mcp

MCP client configuration (e.g. claude_desktop_config.json):

{
  "mcpServers": {
    "ollamadiffuser": {
      "command": "ollamadiffuser-mcp"
    }
  }
}

Works with mcp 1.x and 2.x.

For Claude Code: claude mcp add ollamadiffuser -- ollamadiffuser mcp.

Available MCP tools:

• generate_image -- Generate images from text prompts (auto-loads model; takes a scheduler and a lora). Returns the saved file's path as text and the image itself; when an ollamadiffuser run server already holds the model, that server does the work
• edit_image -- Edit a picture by instruction: the picture's path, more references, a mask, a strength
• generate_video -- Make a short video, from a prompt, a first frame or an audio track; returns the mp4's path
• list_servers -- The OllamaDiffuser servers running on this machine and the model each holds
• list_models -- List available and installed models
• load_model -- Load a model into memory (says why when it won't fit)
• get_status -- Check device, loaded model, and system status

Using OllamaDiffuser from agents

An agent with a shell (Claude Code, Codex, OpenClaw, a script) needs no server and no MCP:

ollamadiffuser generate "a poster that reads 海边小屋" -m qwen-image-2.1-turbo-mlx -o poster.png --json
# {"path": "/…/poster.png", "kind": "image", "model": "qwen-image-2.1-turbo-mlx", "seed": 1873, "elapsed_s": 36.2, "server": null}

• One command, one file. generate, edit and video print the file they wrote. With --json, stdout is one JSON object, or {"error": …} with exit code 1, and everything else goes to stderr.
• No second copy. When a server already holds the model (ollamadiffuser run, the ComfyUI nodes, another agent), the job goes to it. When none does but one was started with ollamadiffuser serve, that one loads the model, so the next job finds it warm. Otherwise the model loads for the job and is let go after it.
• The GPU on another machine. --server 192.168.0.21:8010 sends the job to the OllamaDiffuser server running on that machine. Set it once with OLLAMADIFFUSER_SERVER instead; ollamadiffuser mcp reads it too. The pictures to edit go up from this machine, the result is saved on this machine, and nothing loads here. Start that server with ollamadiffuser run MODEL --host 0.0.0.0 --port 8010, because the default localhost takes no calls from other machines. ollamadiffuser ps --host 192.168.0.21 lists what runs there.
• Seeing what runs. ollamadiffuser ps --json lists every server on this machine and the models it holds, including servers started by older versions or from another config directory. list --json lists installed models with their memory needs, and stop --port N / --all stops servers.
• Memory. On Apple Silicon, where the GPU shares the machine's memory, a model is not loaded when less than its minimum is free. The error gives the numbers. --force (or OLLAMADIFFUSER_SKIP_MEMORY_CHECK=1) loads it anyway.
• A ready-made skill for Claude Code and other skill-aware agents is at integrations/claude-code/ollamadiffuser/SKILL.md. Copy the folder to ~/.claude/skills/.

OpenClaw AgentSkill

An OpenClaw skill is included at integrations/openclaw/SKILL.md. It uses the REST API with response_format=b64_json for agent-friendly base64 image responses. Copy the skill directory to your OpenClaw skills folder or publish to ClawHub.

ComfyUI Nodes

Three custom nodes at integrations/comfyui/ bring OllamaDiffuser's models into a ComfyUI graph — Text to Image, Edit Image (up to four references) and Video (first frame, audio, a control video to follow, LoRA). ComfyUI can't read the MLX builds, so the nodes don't load anything: they call OllamaDiffuser's HTTP API and hand back an ordinary IMAGE or VIDEO for the rest of the graph. A server already running the model is reused; otherwise one is started, after ComfyUI's own models are unloaded, and stopped again afterwards unless keep_loaded is on.

ln -s /path/to/ollamadiffuser/integrations/comfyui ~/ComfyUI/custom_nodes/comfyui-ollamadiffuser

The nodes live in the GitHub repository, not in the pip package — clone the repo, or download the folder, to install them.

Base64 JSON API Response

For AI agents and messaging platforms, use response_format=b64_json to get images as JSON:

curl -X POST http://localhost:8000/api/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt": "a sunset over mountains", "response_format": "b64_json"}'

Response: {"image": "<base64 PNG>", "format": "png", "width": 1024, "height": 1024}

Python API

from ollamadiffuser.core.models.manager import model_manager

# Load model
success = model_manager.load_model("stable-diffusion-1.5")
if success:
    engine = model_manager.loaded_model

    # Text-to-image (seed is optional; omit for random)
    image = engine.generate_image(
        prompt="a beautiful sunset",
        width=1024,
        height=1024,
        seed=42,
    )
    image.save("output.jpg")

    # Image-to-image
    from PIL import Image
    input_img = Image.open("photo.jpg")
    result = engine.generate_image(
        prompt="watercolor painting",
        image=input_img,
        strength=0.7,
    )
    result.save("img2img_output.jpg")
else:
    print("Failed to load model")

📦 Model Ecosystem

Base Models

• Stable Diffusion 1.5: Classic, reliable, fast (img2img + inpainting)
• Stable Diffusion XL: High-resolution, detailed (img2img + inpainting, scheduler overrides)
• Stable Diffusion 3.5: Medium, Large, and Large Turbo variants
• FLUX.1: schnell, dev, Fill, Canny, Depth pipeline variants
• HiDream: Multi-prompt generation with bfloat16
• AnimateDiff: Video/animation generation

Next-Generation Models

• FLUX.2: 32B dev and 4B Klein variants from Black Forest Labs
• Chinese Models: CogView4 (Zhipu), Kolors (Kuaishou), Hunyuan-DiT (Tencent), Z-Image (Alibaba)
• Efficient Models: SANA 1.5 (1.6B), PixArt-Sigma (0.6B) -- high quality at low VRAM
• Open Models: AuraFlow (6.8B, Apache 2.0), OmniGen (3.8B, MIT), Lumina 2.0 (2B, Apache 2.0)

Fast / Turbo Models

• SDXL Turbo: Single-step inference from Stability AI
• SDXL Lightning: 4-step single-file checkpoint from ByteDance (6.5 GB download)
• Z-Image Turbo: 8-step turbo from Alibaba

Community Fine-Tunes

• RealVisXL V4: Photorealistic SDXL, very popular
• DreamShaper: Versatile artistic SD 1.5 model
• Realistic Vision V6: Portrait specialist

GGUF Quantized Models

• FLUX.1-dev GGUF: 7 quantization levels (3GB-16GB VRAM)
• Memory Efficient: Run high-quality models on budget hardware
• Optional Install: ollamadiffuser enable gguf

ControlNet Models

• SD 1.5 ControlNet: 4 control types (canny, depth, openpose, scribble)
• SDXL ControlNet: 2 control types (canny, depth)

LoRA Support

• Hugging Face Integration: Direct download from HF Hub
• Local LoRA Files: Support for local .safetensors files
• Dynamic Loading: Load/unload without model restart
• Strength Control: Adjustable influence (0.1-2.0)

⚙️ Architecture

Strategy Pattern Engine

Each model type has a dedicated strategy class handling loading and generation:

InferenceEngine (facade)
  -> SD15Strategy            (512x512, float16 on MPS, img2img, inpainting)
  -> SDXLStrategy            (1024x1024, float16 on MPS, diffusers force_upcast, img2img, inpainting, scheduler overrides, single-file)
  -> FluxStrategy            (schnell/dev/Fill/Canny/Depth, bfloat16 on MPS, dynamic pipeline class)
  -> SD3Strategy             (1024x1024, float16 on MPS, 28 steps, guidance=3.5)
  -> ControlNetStrategy      (SD15 + SDXL, float16 on MPS, SDXL uses diffusers force_upcast)
  -> VideoStrategy           (AnimateDiff, float16 on MPS, 16 frames)
  -> HiDreamStrategy         (bfloat16 on MPS, multi-prompt)
  -> GGUFStrategy            (quantized via stable-diffusion-cpp)
  -> GenericPipelineStrategy (any diffusers pipeline via config, per-model dtype on MPS, opt-in VAE upcast)

The GenericPipelineStrategy dynamically loads any diffusers pipeline class specified in the model registry, so new models can be added with zero code changes.

Configuration

Models are automatically configured with optimal settings:

• Memory Optimization: Attention slicing, CPU offloading
• Device Detection: Automatic CUDA/MPS/CPU selection
• Precision Handling: FP16/BF16 per model type
• Safety Disabled: Unified SAFETY_DISABLED_KWARGS (no monkey-patching)
• Smart Downloads: Pipeline-only filtering by model type — skips ONNX, Flax, root checkpoints, and safety_checker

🔧 Advanced Usage

ControlNet Parameters

# Fine-tune ControlNet behavior
image = engine.generate_image(
    prompt="architectural masterpiece",
    control_image=control_img,
    controlnet_conditioning_scale=1.2,  # Strength (0.0-2.0)
    control_guidance_start=0.0,         # When to start (0.0-1.0)
    control_guidance_end=1.0            # When to end (0.0-1.0)
)

GGUF Model Usage

# Check GGUF support
ollamadiffuser registry check-gguf

# Download GGUF model for your hardware
ollamadiffuser pull flux.1-dev-gguf-q4ks  # 6GB VRAM
ollamadiffuser pull flux.1-dev-gguf-q3ks  # 4GB VRAM

# Use with optimized settings
ollamadiffuser run flux.1-dev-gguf-q4ks

Batch Processing

from ollamadiffuser.core.utils.controlnet_preprocessors import controlnet_preprocessor

# Pre-initialize for faster processing
controlnet_preprocessor.initialize()

# Process multiple images
prompt = "beautiful landscape"  # Define the prompt
for i, image_path in enumerate(image_list):
    control_img = controlnet_preprocessor.preprocess(image_path, "canny")
    result = engine.generate_image(prompt, control_image=control_img)
    result.save(f"output_{i}.jpg")

API Integration

import requests

# Initialize ControlNet preprocessors
response = requests.post("http://localhost:8000/api/controlnet/initialize")

# Check available preprocessors
response = requests.get("http://localhost:8000/api/controlnet/preprocessors")
print(response.json()["available_types"])

# Generate with file upload
with open("control.jpg", "rb") as f:
    response = requests.post(
        "http://localhost:8000/api/generate/controlnet",
        data={"prompt": "beautiful landscape"},
        files={"control_image": f}
    )

📚 Documentation & Guides

• GGUF Models Guide: Complete guide to memory-efficient GGUF models
• ControlNet Guide: Comprehensive ControlNet usage and examples
• Website Documentation: Complete tutorials and guides

🚀 Performance & Hardware

Minimum Requirements

• RAM: 8GB system RAM
• Storage: 10GB free space
• Python: 3.10+

Recommended Hardware

For Regular Models

• GPU: 8GB+ VRAM (NVIDIA/AMD)
• RAM: 16GB+ system RAM
• Storage: SSD with 50GB+ free space

For Apple Silicon (Mac Mini / MacBook)

• 16GB unified memory: SANA 1.5, Lumina 2.0, DreamShaper, SD 1.5, SDXL/SDXL Turbo, GGUF q2k-q5ks
• 24GB+ unified memory: CogView4, Hunyuan-DiT, FLUX.1-schnell, GGUF q6k-q8
• 32GB unified memory: Kolors, SD 3.5 Large, all MPS-supported models
• GGUF with Metal: ollamadiffuser enable gguf (sets CMAKE_ARGS=-DSD_METAL=ON for GPU acceleration automatically)
• Note: CPU offload does not help on Apple Silicon (unified memory) -- the full model must fit in RAM
• Run ollamadiffuser recommend to see what fits your hardware

For GGUF Models (Memory Efficient)

• GPU: 3GB+ VRAM (or CPU only)
• RAM: 8GB+ system RAM (16GB+ for CPU inference)
• Storage: SSD with 20GB+ free space

Supported Platforms

• CUDA: NVIDIA GPUs (recommended)
• MPS: Apple Silicon (M1/M2/M3/M4) -- native support for 30+ models including GGUF
• CPU: All platforms (slower but functional)

🔧 Troubleshooting

Installation Issues

Missing Dependencies (cv2/OpenCV Error)

The default install already includes OpenCV (it ships as a prebuilt wheel), so this is rare. If you somehow hit ModuleNotFoundError: No module named 'cv2':

# Verify and repair the install
ollamadiffuser verify-deps

# Or install OpenCV directly
pip install "opencv-python>=4.8.0"

GGUF Support Issues

# Enable the GGUF backend (compiles once; sets build flags for you)
ollamadiffuser enable gguf

# Check GGUF support
ollamadiffuser registry check-gguf

# See GGUF_GUIDE.md for detailed troubleshooting

Complete Dependency Check

# Run comprehensive system diagnostics
ollamadiffuser doctor

# Verify and install missing dependencies interactively
ollamadiffuser verify-deps

Clean Installation

If you're having persistent issues, the isolated curl | sh installer sidesteps any conflicts in your existing Python environment:

# Fully isolated reinstall (never touches your system Python)
rm -rf ~/.ollamadiffuser
curl -fsSL https://raw.githubusercontent.com/LocalKinAI/ollamadiffuser/main/install.sh | sh

Or, staying in your own environment:

pip uninstall ollamadiffuser
pip install --no-cache-dir ollamadiffuser   # compile-free core
ollamadiffuser enable gguf   # add optional backends only if you need them
ollamadiffuser verify-deps

Common Issues

Slow Startup

If you experience slow startup, ensure you're using the latest version with lazy loading:

git pull origin main
pip install -e .

ControlNet Not Working

# Check preprocessor status
python -c "
from ollamadiffuser.core.utils.controlnet_preprocessors import controlnet_preprocessor
print('Available:', controlnet_preprocessor.is_available())
print('Initialized:', controlnet_preprocessor.is_initialized())
"

# Manual initialization
curl -X POST http://localhost:8000/api/controlnet/initialize

Memory Issues

# Use GGUF models for lower memory usage
ollamadiffuser pull flux.1-dev-gguf-q4ks  # 6GB VRAM
ollamadiffuser pull flux.1-dev-gguf-q3ks  # 4GB VRAM

# Use smaller image sizes via API
curl -X POST http://localhost:8000/api/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt": "test", "width": 512, "height": 512}' \
  --output test.png

# CPU offloading is automatic
# Close other applications to free memory
# Use basic preprocessors instead of advanced ones

# MLX models (Apple Silicon) hold MLX's buffer cache to 2 GB; on a Mac with
# memory to spare, a larger bound (in GB) lets MLX reuse more of it
OLLAMADIFFUSER_MLX_CACHE_LIMIT_GB=8 ollamadiffuser run qwen-image-2.1-turbo-mlx

Platform-Specific Issues

macOS Apple Silicon

# If you encounter OpenCV issues on Apple Silicon
pip uninstall opencv-python
pip install "opencv-python-headless>=4.8.0"

# GGUF with Metal acceleration — `enable gguf` sets CMAKE_ARGS=-DSD_METAL=ON for you
ollamadiffuser enable gguf

Windows

# If you encounter build errors
pip install --only-binary=all "opencv-python>=4.8.0"

# GGUF on CPU: `ollamadiffuser enable gguf`
# GGUF with CUDA acceleration (advanced — set the flag yourself):
CMAKE_ARGS="-DSD_CUDA=ON" pip install stable-diffusion-cpp-python

Linux

# If you need system dependencies
sudo apt-get update
sudo apt-get install libgl1-mesa-glx libglib2.0-0
pip install opencv-python>=4.8.0

Debug Mode

# Enable verbose logging
ollamadiffuser --verbose run model-name

🤝 Contributing

We welcome contributions! Please check the GitHub repository for contribution guidelines.

🤝 Community & Support

Quick Actions

• 🐛 Report a Bug - Found an issue? Let us know
• 💡 Feature Request - Have an idea? Share it with us
• 💬 Join Discussions - Community discussion
• ⭐ Star on GitHub - Show your support

Community Driven

OllamaDiffuser is an open-source project that thrives on community feedback. Every suggestion, bug report, and contribution helps make it better for everyone.

Open Source • Community Driven • Actively Maintained

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Stability AI: For Stable Diffusion models
• Black Forest Labs: For FLUX.1 and FLUX.2 models
• Alibaba (Tongyi-MAI): For Z-Image Turbo
• NVIDIA (Efficient-Large-Model): For SANA 1.5
• Zhipu AI (THUDM): For CogView4
• Kuaishou (Kwai-Kolors): For Kolors
• Tencent (Hunyuan): For Hunyuan-DiT
• Alpha-VLLM: For Lumina 2.0
• PixArt-alpha: For PixArt-Sigma
• Fal: For AuraFlow
• BAAI (Shitao): For OmniGen
• ByteDance: For SDXL Lightning
• city96: For FLUX.1-dev GGUF quantizations
• Hugging Face: For model hosting and diffusers library
• Anthropic: For Model Context Protocol (MCP)
• OpenClaw: For AI agent ecosystem integration
• ControlNet Team: For ControlNet architecture
• Community: For feedback and contributions

📞 Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions

---

Ready to get started? Install from PyPI: pip install ollamadiffuser or visit ollamadiffuser.com 🎨✨