Real-Time Latent Consistency Model

Original repo from https://github.com/radames/Real-Time-Latent-Consistency-Model, many thanks!

This demo showcases Latent Consistency Model (LCM) using Diffusers with a MJPEG stream server. You can read more about LCM + LoRAs with diffusers here.

You need a webcam to run this demo. 🤗

See a collecting with live demos here

Running Locally

You need CUDA and Python 3.10, Node > 19, Mac with an M1/M2/M3 chip or Intel Arc GPU

Install

python -m venv venv
source venv/bin/activate
pip3 install -r server/requirements.txt
cd frontend && npm install && npm run build && cd ..
python server/main.py --reload --pipeline img2imgSDTurbo 

Don't forget to fuild the frontend!!!

cd frontend && npm install && npm run build && cd ..

LCM

Image to Image

python server/main.py --reload --pipeline img2img 

Image to Image ControlNet Canny

python server/main.py --reload --pipeline controlnet 

LCM + LoRa

Using LCM-LoRA, giving it the super power of doing inference in as little as 4 steps. Learn more here or technical report

Image to Image ControlNet Canny LoRa

python server/main.py --reload --pipeline controlnetLoraSD15

or SDXL, note that SDXL is slower than SD15 since the inference runs on 1024x1024 images

python server/main.py --reload --pipeline controlnetLoraSDXL

Available Pipelines

LCM

img2img
txt2img
controlnet
txt2imgLora
controlnetLoraSD15

SD15

controlnetLoraSDXL
txt2imgLoraSDXL

SDXL Turbo

img2imgSDXLTurbo
controlnetSDXLTurbo

SDTurbo

img2imgSDTurbo
controlnetSDTurbo

Segmind-Vega

controlnetSegmindVegaRT
img2imgSegmindVegaRT

Setting environment variables

• --host: Host address (default: 0.0.0.0)
• --port: Port number (default: 7860)
• --reload: Reload code on change
• --max-queue-size: Maximum queue size (optional)
• --timeout: Timeout period (optional)
• --safety-checker: Enable Safety Checker (optional)
• --torch-compile: Use Torch Compile
• --use-taesd / --no-taesd: Use Tiny Autoencoder
• --pipeline: Pipeline to use (default: "txt2img")
• --ssl-certfile: SSL Certificate File (optional)
• --ssl-keyfile: SSL Key File (optional)
• --debug: Print Inference time
• --compel: Compel option
• --sfast: Enable Stable Fast
• --onediff: Enable OneDiff

If you run using bash build-run.sh you can set PIPELINE variables to choose the pipeline you want to run

PIPELINE=txt2imgLoraSDXL bash build-run.sh

and setting environment variables

TIMEOUT=120 SAFETY_CHECKER=True MAX_QUEUE_SIZE=4 python server/main.py --reload --pipeline txt2imgLoraSDXL

openssl req -newkey rsa:4096 -nodes -keyout key.pem -x509 -days 365 -out certificate.pem
python server/main.py --reload --ssl-certfile=certificate.pem --ssl-keyfile=key.pem

Docker

You need NVIDIA Container Toolkit for Docker, defaults to `controlnet``

docker build -t lcm-live .
docker run -ti -p 7860:7860 --gpus all lcm-live

reuse models data from host to avoid downloading them again, you can change ~/.cache/huggingface to any other directory, but if you use hugingface-cli locally, you can share the same cache

docker run -ti -p 7860:7860 -e HF_HOME=/data -v ~/.cache/huggingface:/data  --gpus all lcm-live

or with environment variables

docker run -ti -e PIPELINE=txt2imgLoraSDXL -p 7860:7860 --gpus all lcm-live

NDI Output (optional)

You can broadcast the generated frames as an NDI video source. This requires the NDI SDK and the Python bindings (ndi-python) to be installed on the host.

• Enable from CLI:

python server/main.py --reload --pipeline img2imgSDTurbo --ndi-send --ndi-name "LCM NDI"

• Or via env vars:

NDI_SEND=True NDI_NAME="LCM NDI" python server/main.py --reload --pipeline img2imgSDTurbo

Notes:

• If NDIlib is not importable, the server will continue to run and NDI will be silently disabled.
• The advertised stream name defaults to LCM NDI and can be changed via --ndi-name.
• Install NDI SDK and ndi-python for your platform from the official sources before enabling this feature.

Syphon Output (macOS, optional)

Broadcast frames via syphon-python as a Syphon source (Metal backend).

• Install dependencies:

pip install syphon-python numpy

• Enable from CLI (default vertical flip ON to match OBS expectations):

python server/main.py --reload --pipeline img2imgSDTurbo --syphon-send --syphon-name "LCM Syphon"

• Or via env vars:

SYPHON_SEND=True SYPHON_NAME="LCM Syphon" python server/main.py --reload --pipeline img2imgSDTurbo

Notes:

• The server name will appear in Syphon clients (e.g., OBS Syphon Client) as LCM Syphon <user_id> per session.
• Vertical flip is enabled by default to correct coordinate origin differences. Disable it with --no-syphon-flip-vertical or SYPHON_FLIP_VERTICAL=False.
• Requires macOS 11+.

Demo on Hugging Face

• radames/Real-Time-Latent-Consistency-Model
• radames/Real-Time-SD-Turbo
• latent-consistency/Real-Time-LCM-ControlNet-Lora-SD1.5
• latent-consistency/Real-Time-LCM-Text-to-Image-Lora-SD1.5
• radames/Real-Time-Latent-Consistency-Model-Text-To-Image