SimpleTuner 💹

ℹ️ No data is sent to any third parties except through opt-in flag report_to, push_to_hub, or webhooks which must be manually configured.

SimpleTuner is geared towards simplicity, with a focus on making the code easily understood. This codebase serves as a shared academic exercise, and contributions are welcome.

If you'd like to join our community, we can be found on Discord via Terminus Research Group. If you have any questions, please feel free to reach out to us there.

Table of Contents

• Design Philosophy
• Tutorial
• Features
  • Flux
  • Wan 2.1 Video
  • LTX Video
  • PixArt Sigma
  • NVLabs Sana
  • Stable Diffusion 2.0/2.1
  • Stable Diffusion 3.0
  • Kwai Kolors
  • Lumina2
• Hardware Requirements
  • Flux
  • SDXL
  • Stable Diffusion (Legacy)
• Scripts
• Toolkit
• Setup
• Troubleshooting

Design Philosophy

• Simplicity: Aiming to have good default settings for most use cases, so less tinkering is required.
• Versatility: Designed to handle a wide range of image quantities - from small datasets to extensive collections.
• Cutting-Edge Features: Only incorporates features that have proven efficacy, avoiding the addition of untested options.

Tutorial

Please fully explore this README before embarking on the tutorial, as it contains vital information that you might need to know first.

For a quick start without reading the full documentation, you can use the Quick Start guide.

For memory-constrained systems, see the DeepSpeed document which explains how to use 🤗Accelerate to configure Microsoft's DeepSpeed for optimiser state offload.

For multi-node distributed training, this guide will help tweak the configurations from the INSTALL and Quickstart guides to be suitable for multi-node training, and optimising for image datasets numbering in the billions of samples.

---

Features

• Multi-GPU training
• New token-wise dropout techniques like TREAD for speeding up Wan 2.1/2.2 and Flux training, including Kontext
• Image, video, and caption features (embeds) are cached to the hard drive in advance, so that training runs faster and with less memory consumption
• Aspect bucketing: support for a variety of image/video sizes and aspect ratios, enabling widescreen and portrait training.
• Refiner LoRA or full u-net training for SDXL
• Most models are trainable on a 24G GPU, or even down to 16G at lower base resolutions.
  • LoRA/LyCORIS training for PixArt, SDXL, SD3, and SD 2.x that uses less than 16G VRAM
• DeepSpeed integration allowing for training SDXL's full u-net on 12G of VRAM, albeit very slowly.
• Quantised NF4/INT8/FP8 LoRA training, using low-precision base model to reduce VRAM consumption.
• Optional EMA (Exponential moving average) weight network to counteract model overfitting and improve training stability.
• Train directly from an S3-compatible storage provider, eliminating the requirement for expensive local storage. (Tested with Cloudflare R2 and Wasabi S3)
• For SDXL, SD 1.x/2.x, and Flux, full or LoRA based ControlNet model training (not ControlLite)
• Training Mixture of Experts for lightweight, high-quality diffusion models
• Masked loss training for superior convergence and reduced overfitting on any model
• Strong prior regularisation training support for LyCORIS models
• Webhook support for updating eg. Discord channels with your training progress, validations, and errors
• Integration with the Hugging Face Hub for seamless model upload and nice automatically-generated model cards.
  • Use the datasets library (more info) to load compatible datasets directly from the hub

HiDream

Full training support for HiDream is included:

• Custom ControlNet implementation for training via full-rank, LoRA or Lycoris
• Memory-efficient training for NVIDIA GPUs (AMD support is planned)
• Dev and Full both functioning and trainable. Fast is untested.
• Optional MoEGate loss augmentation
• Lycoris or full tuning via DeepSpeed ZeRO on a single GPU
• Quantise the base model using --base_model_precision to int8-quanto or fp8-quanto for major memory savings
• Quantise Llama LLM using --text_encoder_4_precision set to int4-quanto or int8-quanto to run on 24G cards.

See hardware requirements or the quickstart guide.

Flux.1

Full training support for Flux.1 is included:

• Double the training speed of Flux.1 with the new --fuse_qkv_projections option, taking advantage of Flash Attention 3 on Hopper systems
• ControlNet training via full-rank, LoRA or Lycoris
• Instruct fine-tuning for the Kontext [dev] editing model implementation generously provided by Runware.
• Classifier-free guidance training
  • Leave it disabled and preserve the dev model's distillation qualities
  • Or, reintroduce CFG to the model and improve its creativity at the cost of inference speed and training time.
• (optional) T5 attention masked training for superior fine details and generalisation capabilities
• LoRA or full tuning via DeepSpeed ZeRO on a single GPU
• Quantise the base model using --base_model_precision to int8-quanto or fp8-torchao for major memory savings

See hardware requirements or the quickstart guide.

Wan Video

SimpleTuner has preliminary training integration for Wan 2.1 which has a 14B and 1.3B type, both of which work.

• Text to Video training is supported.
• Image to Video training is not yet supported.
• Text encoder training is not supported.
• VAE training is not supported.
• LyCORIS, PEFT, and full tuning all work as expected
• ControlNet training is not yet supported

See the Wan Video Quickstart guide to start training.

LTX Video

SimpleTuner has preliminary training integration for LTX Video, efficiently training on less than 16G.

• Text encoder training is not supported
• VAE training is not supported
• LyCORIS, PEFT, and full tuning all work as expected
• ControlNet training is not yet supported

See the LTX Video Quickstart guide to start training.

PixArt Sigma

SimpleTuner has extensive training integration with PixArt Sigma - both the 600M & 900M models load without modification.

• Text encoder training is not supported
• LyCORIS and full tuning both work as expected
• ControlNet training is supported for full and PEFT LoRA training
• Two-stage PixArt training support (see: MIXTURE_OF_EXPERTS)

See the PixArt Quickstart guide to start training.

NVLabs Sana

SimpleTuner has extensive training integration with NVLabs Sana.

This is a lightweight, fun, and fast model that makes getting into model training highly accessible to a wider audience.

• LyCORIS and full tuning both work as expected.
• Text encoder training is not supported.
• PEFT Standard LoRA is not supported.
• ControlNet training is not yet supported

See the NVLabs Sana Quickstart guide to start training.

Stable Diffusion 3

• LoRA and full finetuning are supported as usual.
• ControlNet training via full-rank, PEFT LoRA, or Lycoris
• Certain features such as segmented timestep selection and Compel long prompt weighting are not yet supported.
• Parameters have been optimised to get the best results, validated through from-scratch training of SD3 models

See the Stable Diffusion 3 Quickstart to get going.

Kwai Kolors

An SDXL-based model with ChatGLM (General Language Model) 6B as its text encoder, doubling the hidden dimension size and substantially increasing the level of local detail included in the prompt embeds.

Kolors support is almost as deep as SDXL, minus ControlNet training support.

Lumina2

A 2B parameter flow-matching model that uses the 16ch Flux VAE.

• LoRA, Lycoris, and full finetuning are supported
• ControlNet training is not yet supported

A Lumina2 Quickstart is available with example configurations.

Cosmos2 Predict (Image)

A 2B / 14B parameter model that can do video as well as text-to-image.

• Currently, only the text-to-image variant is supported.
• Lycoris or full-rank tuning are supported, but PEFT LoRAs are currently not.
• ControlNet training is not yet supported.

A Cosmos2 Predict Quickstart is available with full example configuration and dataset.

Qwen-Image

A massive 20B MMDiT that can do text-to-image.

• Lycoris, LoRA, and full-rank training are all supported, with full-rank training requiring H200 or better with DeepSpeed
• ControlNet training is not yet supported.

A Qwen Image Quickstart is available with example configuration and dataset, as well as general training/configuration tips.

Legacy Stable Diffusion models

RunwayML's SD 1.5 and StabilityAI's SD 2.x are both trainable under the legacy designation.

---

Hardware Requirements

NVIDIA

Pretty much anything 3080 and up is a safe bet. YMMV.

AMD

LoRA and full-rank tuning are verified working on a 7900 XTX 24GB and MI300X.

Lacking xformers, it will use more memory than Nvidia equivalent hardware.

Apple

LoRA and full-rank tuning are tested to work on an M3 Max with 128G memory, taking about 12G of "Wired" memory and 4G of system memory for SDXL.

• You likely need a 24G or greater machine for machine learning with M-series hardware due to the lack of memory-efficient attention.
• Subscribing to Pytorch issues for MPS is probably a good idea, as random bugs will make training stop working.

HiDream [dev, full]

• A100-80G (Full tune with DeepSpeed)
• A100-40G (LoRA, LoKr)
• 3090 24G (LoRA, LoKr)

HiDream has not been tested on 16G cards, but with aggressive quantisation and pre-caching of embeds, you might make it work, though even 24G is pushing limits.

Flux.1 [dev, schnell]

• A100-80G (Full tune with DeepSpeed)
• A100-40G (LoRA, LoKr)
• 3090 24G (LoRA, LoKr)
• 4060 Ti 16G, 4070 Ti 16G, 3080 16G (int8, LoRA, LoKr)
• 4070 Super 12G, 3080 10G, 3060 12GB (nf4, LoRA, LoKr)

Flux prefers being trained with multiple large GPUs but a single 16G card should be able to do it with quantisation of the transformer and text encoders.

Kontext requires a bit beefier compute and memory allocation; a 4090 will go from ~3 to ~6 seconds per step when it is enabled.

Auraflow

• A100-80G (Full tune with DeepSpeed)
• A100-40G (LoRA, LoKr)
• 3090 24G (LoRA, LoKr)
• 4060 Ti 16G, 4070 Ti 16G, 3080 16G (int8, LoRA, LoKr)
• 4070 Super 12G, 3080 10G, 3060 12GB (nf4, LoRA, LoKr)

SDXL, 1024px

• A100-80G (EMA, large batches, LoRA @ insane batch sizes)
• A6000-48G (EMA@768px, no EMA@1024px, LoRA @ high batch sizes)
• A100-40G (EMA@1024px, EMA@768px, EMA@512px, LoRA @ high batch sizes)
• 4090-24G (EMA@1024px, batch size 1-4, LoRA @ medium-high batch sizes)
• 4080-12G (LoRA @ low-medium batch sizes)

Stable Diffusion 2.x, 768px

• 16G or better

Toolkit

For more information about the associated toolkit distributed with SimpleTuner, refer to the toolkit documentation.

Setup

Detailed setup information is available in the installation documentation.

Troubleshooting

Enable debug logs for a more detailed insight by adding export SIMPLETUNER_LOG_LEVEL=DEBUG to your environment (config/config.env) file.

For performance analysis of the training loop, setting SIMPLETUNER_TRAINING_LOOP_LOG_LEVEL=DEBUG will have timestamps that highlight any issues in your configuration.

For a comprehensive list of options available, consult this documentation.