Z-Image ncnn Vulkan

:exclamation: :exclamation: :exclamation: This software is in the early development stage, it may bite your cat

ncnn implementation of Z-Image image generator, with LanPaint inpaint/outpaint and ControlNet support.

zimage-ncnn-vulkan uses ncnn project as the universal neural network inference framework.

Download

Download Windows/Linux/macOS Executable for Intel/AMD/NVIDIA/Apple-Silicon GPU

https://github.com/nihui/zimage-ncnn-vulkan/releases

This package includes all the binaries required. It is portable, so no CUDA or PyTorch runtime environment is needed :)

prepare model files

Download the z-image-turbo and z-image model folders to the same directory as the executable file

https://huggingface.co/nihui-szyl/z-image-ncnn/tree/main

For ControlNet, download the z-image-control model folder to the same directory as z-image-turbo. For Tile ControlNet upscale, download the z-image-control-tile model folder too.

zimage-ncnn-vulkan
z-image-turbo/
z-image-control/
z-image-control-tile/

About Z-Image

Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer

https://github.com/Tongyi-MAI/Z-Image

Usages

requirements

• Minimum (Linux / macOS): 16GB RAM, any Vulkan capable GPU

• Minimum (Windows):

  Due to WDDM limitations: Vulkan applications can only use half of the system RAM.

  The following condition shall be met: (Half of system RAM) + (GPU memory) >= 16GB

  Examples of valid combinations:

  • Any amount RAM, 16GB dedicated GPU
  • 16GB RAM, 8GB dedicated GPU
  • 24GB RAM, 4GB dedicated GPU
  • 32GB RAM, any Vulkan capable GPU

• Recommended: 32GB RAM, 16GB dedicated GPU with tensorcore/matrix hardware

Example Command

Text to image

zimage-ncnn-vulkan.exe -p "風的彷徨." -o output.png

LanPaint inpaint

zimage-ncnn-vulkan.exe -i input.png -k mask.png -p "a cute girl" -o output.png

LanPaint outpaint

zimage-ncnn-vulkan.exe -i input.png -x 128,128,128,128 -p "a cute girl" -o output.png

ControlNet

zimage-ncnn-vulkan.exe -c pose.png -p "a cute girl, full body" -o output.png

Tile ControlNet upscale

zimage-ncnn-vulkan.exe -c lowres.png -t -s 2048,2048 -p "a cute girl" -o output.png

Full Usages

Usage: zimage-ncnn-vulkan -p prompt -o outfile [options]...

  -h                   show this help
  -p prompt            prompt (default=rand)
  -n negative-prompt   negative prompt (optional)
  -o output-path       output image path (default=out.png)
  -i input-image       input image for inpaint (optional)
  -k mask-image        inpaint mask, white=paint black=preserve (optional)
  -x l,t,r,b           outpaint by expanding input canvas (optional)
  -c control-image     control image for ControlNet (optional)
  -w control-scale     ControlNet scale (default=1.0)
  -t                   Tile ControlNet upscale mode (optional)
  -s image-size        image resolution (default=1024,1024)
  -l steps             denoise steps (default=auto)
  -r random-seed       random seed (default=rand)
  -m model-path        z-image model path (default=z-image-turbo)
  -g gpu-id            gpu device to use (-1=cpu, default=auto)
  -b batch-size        batched generation (default=1)

LanPaint inpaint/outpaint requires -i input-image. Inpaint uses -k mask-image, where white means repaint and black means preserve. Outpaint uses -x left,top,right,bottom.

ControlNet uses -c control-image. The control image size must match the output image size. ControlNet currently requires the z-image-turbo model and the z-image-control model folder.

Tile ControlNet upscale uses -c lowres-image -t -s target-width,target-height. The low resolution image is resized internally to the target size and used as tile control. It loads ControlNet weights from z-image-control-tile.

If you encounter a crash or error, try upgrading your GPU driver:

• Intel: https://downloadcenter.intel.com/product/80939/Graphics-Drivers
• AMD: https://www.amd.com/en/support
• NVIDIA: https://www.nvidia.com/Download/index.aspx

Build from Source

1. Clone this project with all submodules

git clone https://github.com/nihui/zimage-ncnn-vulkan.git
cd zimage-ncnn-vulkan
git submodule update --init --recursive --depth 1

2. Build with CMake

mkdir build
cd build
cmake ../src
cmake --build . -j 4

Sample Images

LanPaint inpaint

LanPaint outpaint

ControlNet pose

ControlNet canny

ControlNet gray

ControlNet tile low resolution

Original Z-Image Project

• https://github.com/Tongyi-MAI/Z-Image

Other Open-Source Projects Referenced

• https://github.com/scraed/LanPaint for inpaint and outpaint pipeline
• https://huggingface.co/alibaba-pai/Z-Image-Turbo-Fun-Controlnet-Union-2.1 for Z-Image ControlNet model

Other Open-Source Code Used

• https://github.com/Tencent/ncnn for fast neural network inference on ALL PLATFORMS
• https://github.com/futz12/ncnn_llm for BPE tokenizer
• https://github.com/webmproject/libwebp for encoding and decoding Webp images on ALL PLATFORMS
• https://github.com/libjpeg-turbo/libjpeg-turbo for encoding and decoding JPEG images on ALL PLATFORMS
• https://github.com/pnggroup/libpng for encoding and decoding PNG images on ALL PLATFORMS
• https://github.com/zlib-ng/zlib-ng for encoding and decoding PNG images on ALL PLATFORMS
• https://github.com/tronkko/dirent for listing files in directory on Windows