DFloat11 Compressed Model: black-forest-labs/FLUX.1-Krea-dev

This is a DFloat11 losslessly compressed version of the original black-forest-labs/FLUX.1-Krea-dev model. It reduces model size by 32% compared to the original BFloat16 model, while maintaining bit-identical outputs and supporting efficient GPU inference.

🔥🔥🔥 Thanks to DFloat11 compression, FLUX.1-Krea-dev can now run on a single 24GB GPU, or on a 12GB GPU with CPU offloading, while maintaining full model quality. 🔥🔥🔥

📊 Performance Comparison

🔧 How to Use

1. Install or upgrade the DFloat11 pip package (installs the CUDA kernel automatically; requires a CUDA-compatible GPU and PyTorch installed):

   pip install -U dfloat11[cuda12]
   

2. Install or upgrade diffusers:

   pip install -U diffusers
   

3. Save the following code to a Python file krea.py:

   import argparse
   import time
   
   import torch
   from diffusers import FluxPipeline
   from dfloat11 import DFloat11Model
   
   # Parse command line arguments
   parser = argparse.ArgumentParser(description="Generate images using FLUX.1-Krea-dev model")
   parser.add_argument(
       "--prompt", type=str, help="Text prompt for image generation",
       default="An astronaut, helmet off, sits at a tiny table set on the tip of a crescent moon, sipping tea while gazing at a swirling galaxy in the distance. Stars twinkle around, casting a gentle glow on the lunar surface.",
   )
   parser.add_argument("--width", type=int, default=1024, help="Image width")
   parser.add_argument("--height", type=int, default=1024, help="Image height")
   parser.add_argument("--guidance_scale", type=float, default=4.5, help="Guidance scale for generation")
   parser.add_argument("--save_file_name", type=str, default="flux-krea-dev.png", help="Output file name")
   parser.add_argument("--cpu_offload", action="store_true", help="Enable DFloat11 CPU offloading")
   args = parser.parse_args()
   
   # Load the pipeline
   pipe = FluxPipeline.from_pretrained(
       "black-forest-labs/FLUX.1-Krea-dev",
       torch_dtype=torch.bfloat16,
   )
   
   # Load DFloat11 model
   DFloat11Model.from_pretrained(
       "DFloat11/FLUX.1-Krea-dev-DF11",
       bfloat16_model=pipe.transformer,
       device="cpu",
       cpu_offload=args.cpu_offload,
   )
   pipe.enable_model_cpu_offload()
   
   start_time = time.time()
   # Generate image
   image = pipe(
       args.prompt,
       height=args.height,
       width=args.width,
       guidance_scale=args.guidance_scale,
   ).images[0]
   end_time = time.time()
   
   # Save the image
   image.save(args.save_file_name)
   
   # Print time and memory usage
   print(f"Time taken: {end_time - start_time:.2f} seconds")
   peak_memory = torch.cuda.max_memory_allocated()
   print(f"Peak memory: {peak_memory / 1000 ** 3:.2f} GB")
   

4. To run without CPU offloading (18GB VRAM required):

   python krea.py
   

   To run with CPU offloading (10GB VRAM required):

   python krea.py --cpu_offload
   

🔍 How It Works

We apply Huffman coding to losslessly compress the exponent bits of BFloat16 model weights, which are highly compressible (their 8 bits carry only ~2.6 bits of actual information). To enable fast inference, we implement a highly efficient CUDA kernel that performs on-the-fly weight decompression directly on the GPU.

The result is a model that is ~32% smaller, delivers bit-identical outputs, and achieves performance comparable to the original BFloat16 model.

Learn more in our research paper.

📄 Learn More

• Paper: 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float
• GitHub: https://github.com/LeanModels/DFloat11
• HuggingFace: https://huggingface.co/DFloat11