gguf quantized version of kontext

• run it straight with gguf-connector
• opt a gguf file in the current directory to interact with by:

ggc k0

GGUF file(s) available. Select which one to use:

1. flux-kontext-lite-q2_k.gguf
2. flux-kontext-lite-q4_0.gguf
3. flux-kontext-lite-q8_0.gguf

Enter your choice (1 to 3): _

note: try experimental lite model with 8-step operation; save up to 70% loading time

run it with gguf-node via comfyui

• drag kontext to > ./ComfyUI/models/diffusion_models
• drag clip-l, t5xxl to > ./ComfyUI/models/text_encoders
• drag pig to > ./ComfyUI/models/vae

• don't need safetensors anymore; all gguf (model + encoder + vae)
• full set gguf works on gguf-node (see the last item from reference at the very end)
• get more t5xxl gguf encoder either here or here

extra: scaled safetensors (alternative 1)

• get all-in-one checkpoint here (model, clips and vae embedded)
• another option: get multi matrix scaled fp8 from comfyui here or e4m3fn fp8 here with seperate scaled version l-clip, t5xxl and vae

run it with diffusers🧨 (alternative 2)

• might need the most updated diffusers (git version) for FluxKontextPipeline to work; upgrade your diffusers with:

pip install git+https://github.com/huggingface/diffusers.git

• see example inference below:

import torch
from transformers import T5EncoderModel
from diffusers import FluxKontextPipeline
from diffusers.utils import load_image

text_encoder = T5EncoderModel.from_pretrained(
    "calcuis/kontext-gguf",
    gguf_file="t5xxl_fp16-q4_0.gguf",
    torch_dtype=torch.bfloat16,
    )

pipe = FluxKontextPipeline.from_pretrained(
    "calcuis/kontext-gguf",
    text_encoder_2=text_encoder,
    torch_dtype=torch.bfloat16
    ).to("cuda")

input_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png")

image = pipe(
  image=input_image,
  prompt="Add a hat to the cat",
  guidance_scale=2.5
).images[0]
image.save("output.png")

• tip: if your machine doesn't has enough vram, would suggest running it with gguf-node via comfyui (plan a), otherwise you might expect to wait very long while falling to a slow mode; this is always a winner takes all game

run it with gguf-connector (other alternatives)

• simply execute the command below in console/terminal

ggc k2

• note: during the first time launch, it will pull the required model file(s) from this repo to local cache automatically; then opt to run it entirely offline; i.e., from local URL: http://127.0.0.1:7860 with lazy webui

• with bot lora embedded version

ggc k1

• new plushie style

additional chapter for lora conversion via gguf-connector

• convert lora from base to unet format, i.e.,plushie, then it can be used in comfyui as well

ggc la

• able to swap the lora back (from unet to base; auto-detection logic applied), then it can be used for inference again

ggc la

update

• clip-l-v2: missing tensor text_projection.weight added
• kontext-v2: s-quant and k-quant; except single and double blocks, all in f32 status
  • pros: load faster (as no dequant needed for those tensors); and
    1. avoid key breaking issue, since some inference engines only dequant blocks;
    2. compatible for non-cuda machines, as most of them cannot run bf16 tensors
  • cons: little bit larger in file size
• kontext-v3: i-quant attempt (upgrade your node to the latest version for full quant support)
• kontext-v4: t-quant; runnable (extramely fast); for speed test/experimental purposes

not all included in the initial test (*tested with a beginner laptop gpu only, if you have highend model, might find q8_0 running surprisingly faster than others), the rest of them, test it yourself; btw, the interesting thing is: the loading time required was not aligning with file size, due to the complexity of each calculation (dequant), and might vary from models

new memory economy mode

• this option works for machine with low/no vram or even without gpu

ggc k3

🐷 Kontext Image Editor (connector mode) 🐷

• opt a gguf file straight in the current directory to interact with

ggc k6

• semi-full quant supported in the k8 connector (use dequantor instead of diffusers)

ggc k8

reference

• base model from black-forest-labs
• comfyui from comfyanonymous
• gguf-node (pypi|repo|pack)
• gguf-connector (pypi)