exl3-qlora

QLoRA fine-tuning directly on EXL3-quantized models. This repo began as a fork of turboderp's ExLlamaV3 and still contains the full inference library (synced to upstream v1.6.1 — see the upstream README for inference, conversion, and installation documentation). On top of it there is a self-contained, transformers-free training path: a differentiable forward built over the EXL3 trellis (validated against the native inference forward to 100% argmax agreement), a plain-PyTorch trainer, and adapters that save in standard PEFT format.

Why train on EXL3 instead of bitsandbytes?

• Train on 2-8bpw quants, including non-integer. BNB is 4 or 8 bit.
• EXL3 is higher accuracy than BNB, theoretically this should help? - still needs benchmarking
• EXL3 is more performant in the lower and mid-size batches that are often used when squeezing big models onto consumer cards.

Quick start

# install: PyTorch (CUDA 12.4+) first, then this repo (builds the CUDA extension),
# plus the one training dep
pip install -r requirements.txt
pip install .
pip install datasets            # optional extras: flash-attn, liger-kernel, bitsandbytes

# 1. prove the differentiable forward is correct for YOUR model (run this first)
python training/qlora_validate_native.py --model /path/to/exl3-model --compute-dtype bfloat16

# 2. edit the config, then train
python training/qlora_train.py --config training/qlora_train_config.yaml

# 3. before/after comparison on the native inference path
python training/qlora_infer_native.py --model /path/to/exl3-model --adapter out/my_adapter

Everything is driven by one YAML file (training/qlora_train_config.yaml is the fully-commented reference — its keys mirror the CLI flags of training/qlora_train_native.py one-to-one). A minimal config looks like:

model: /models/Llama-3.2-3B-Instruct-exl3-4bpw
out: out/my_adapter
parallel: split           # single | split (layer-split across GPUs) | ddp (torchrun)

r: 64
alpha: 64.0               # use alpha = r with init_lora: pissa
lr: 5e-6
epochs: 1.0
batch: 3
seq_len: 8192
pack: true                # sample packing (best-fit-decreasing, ~98% fill)

dataset: /data/my-set.jsonl
messages_key: messages    # OpenAI-style chats; or instruction/context/response keys
prompt_format: auto

init_lora: pissa          # default | pissa | qerr | eva  (see below)
eval_split: test          # held-out eval from the dataset's own split
eval_every: 50
compute_dtype: bfloat16
use_liger: true
optim: paged_adamw8bit

What's in the box

• Single-GPU, multi-GPU layer-split (parallel: split), and DDP (parallel: ddp) training of LoRA adapters over a frozen EXL3 base — plus optional embedding/LM-head training (full or low-rank).
• Preference optimization: DPO, KTO, and SimPO (training/qlora_train_pref.py --method dpo|kto|simpo, or via the YAML launcher with method: dpo|kto|simpo and the qlora_train_pref_config.yaml template). DPO/KTO use the frozen quantized base as the reference model (adapter-disable trick — no second model copy); SimPO is reference-free (length-normalized rewards + target margin γ — no reference forward, so a step costs about half a DPO step). Loss semantics follow HuggingFace TRL's stable DPOTrainer/KTOTrainer/CPOTrainer (with credit — see below), so β/loss-variant hyperparameters transfer directly; variants: sigmoid/cDPO, hinge (SLiC), IPO, KTO, APO-zero-unpaired, SimPO (+ optional CPO-style SFT mix).
• Real-time (inference-time) training: one loaded model serves generation and trains its adapter in place (exllamav3.training.RealtimeQLoRA) — ingest() a batch of samples, the updated adapter is live in the very next generation (in-memory sync into the runtime LoRA slots; KV cache invalidated on update, or target only non-KV projections and skip invalidation). Constant externally-settable LR, timestamped rollback checkpoints, a readers/writer lock separating serving from training. Interactive demo + server-integration reference: training/realtime_chat.py.
• Memory levers for long context on consumer cards: gradient checkpointing, activation offload to CPU RAM, fused/chunked cross-entropy (chunked over the vocab too, for 256k-vocab models), 8-bit and paged optimizers, Liger kernels (RMSNorm/RoPE/SwiGLU).
• A real eval harness: held-out loss from your dataset's own split (eval_split) or a carved fraction (val_frac), an optional second monitor set (eval2_*, e.g. wikitext LM loss watched next to your task loss), save_best checkpointing, periodic live sample generations, and a per-run CSV log of hyperparameters/losses/VRAM/throughput.
• Correctness gates, not vibes: qlora_validate_native.py checks the differentiable forward against the native inference forward, the Liger backward against plain torch, packing isolation, and each adapter init's step-0 math — before you spend GPU-days. A CPU test suite covers the gradient path end-to-end.
• Vision (image+text) SFT on the VLM bases exllamav3 serves — Qwen2.5/3/3.5-VL, Gemma3/4, Mistral3 (vision: true): the model's own vision tower runs frozen, its features are spliced into the text tower at the image positions with the arch's exact image token layout (3-D mRoPE, deepstack, Gemma4's bidirectional image spans all reproduced), and only the language model's adapters train — Axolotl's recipe, with the same content-parts dataset layout. See Vision training below.
• Standard outputs: adapters save as PEFT-format safetensors, loadable by exllamav3's native LoRA loader (TabbyAPI), PEFT, or merge scripts. Runtime adapters apply correctly on every inference path; note that decode is slower while an adapter is loaded (the graph-fused decode paths can't run under one — unload() restores full speed), and deploying via merge-and-requantize has no hit at all.

Supported architectures (training)

The differentiable forward reads every norm/activation/scale from the loaded modules and is validated against the native inference forward per architecture. Unsupported layouts are rejected loudly at construction — nothing silently mistrains.

MoE note: the plain gate_proj/up_proj/down_proj targets adapt dense MLPs and the always-active shared expert; routed experts are opt-in (expert_gate_proj etc., with --expert-r for rank). Routers stay frozen. On AFMoE the attention gate is keyed self_attn.gate_proj in the checkpoint and rides the gate_proj target (or attn_gate_proj to adapt it alone). Spark-X2.5's headwise gate (self_attn.g_proj, one scalar per head, fp16) is opt-in only: name g_proj or attn_gate_proj in targets to adapt it; the default target list leaves it frozen.

Streaming routed experts from host RAM (16 GB cards)

--stream-experts (YAML stream_experts: true) keeps every routed expert's packed EXL3 weights in page-locked host RAM and copies them into a small VRAM ring one MoE layer at a time, one layer ahead of the compute (exllamav3/training/expert_stream.py). The dense trunk, routers, shared experts, embedding and head stay on the GPU; the model is loaded block by block so the peak during loading is the trunk plus one layer's experts. It is value-exact: the reconstruct kernel reads the same bytes from the ring slot that it would have read from the resident tensor, so a same-seed streamed run gives the same loss as a resident run. The cost is bandwidth: each MoE layer's experts cross PCIe once per pass, three passes per step under gradient checkpointing (two with --dequant-cache) — on Qwen3.8-Flash-Next at 2.05 bpw that is ~31 GB per pass, a few seconds per step on PCIe 4 x16, overlapped with the layer's own compute. --stream-experts-slots N (default 2) prefetches further ahead at one layer's experts of VRAM per extra slot.

Requirements: one GPU (--parallel single), Linux, EXL3-quantized experts, and host RAM for all of them plus the usual working set — checked before loading (Qwen3.8-Flash-Next 2.05bpw: ~31 GB of experts, so >= 64 GB with --ngram-ram, ~48 GB with the n-gram table left on disk). Not combinable with --parallel split, --vram-spillover, routed-expert adapters (expert_* targets, --expert-r) or live samples (--sample-every must be 0: samples run the inference forward, which needs resident experts; for inference on the same card the inference side has its own CPU expert offload, EXL3_MOE_CPU_OFFLOAD in doc/env_vars.md, untested with an adapter here).

Qwen3.8-Flash-Next on a single 16 GB card — untested recipe: nothing on this path has run on a box yet, and the first run is the test (budget per the plan: ~2 GB dense weights, ~1.3 GB for two expert slots, ~1 GB of stream-stack activations offloadable, < 0.5 GB adapters + 8-bit Adam, the 248k-vocab head chunked):

python training/qlora_train_native.py --model /models/Qwen3.8-Flash-Next-exl3-2.05bpw_h4_ng4 \
  --stream-experts --sample-every 0 \
  --targets q_proj k_proj v_proj o_proj z_proj b_proj a_proj gate_proj up_proj down_proj \
  --r 16 --alpha 32 --optim adamw8bit --offload-activations --attn-impl flash \
  --seq-len 512 --batch 1 --grad-accum 4 --steps 30 --lr 1e-4 \
  --head-vocab-chunk 32768 --lora-head \
  --dataset /data/my-set.jsonl --messages-key messages --out out/qwen38-flash-next-lora
# add --ngram-ram with >= 64 GB of host RAM (the n-gram table in RAM instead of streamed from disk)

(q/k/v/o_proj also select the Gated DeltaNet in/out projections; z/b/a_proj its gate/beta/decay projections; gate/up/down_proj the always-active shared expert.) The run prints one expert streaming: ... layers parked ... line after loading and a [STREAM] line at the end with the copy counts; a !! expert streaming: parking layer ... freed ... MiB warning during loading means something still holds the device copies and the budget will not hold. Full plan and status: doc/qwen38_flash_next_plan.md.

Prompt formats (--prompt-format / prompt_format:)

All formats do exact prompt/response boundary masking (prompt and response are tokenized separately) and single-BOS normalization; verify any new base with --inspect 3 before training.

Vision (image + text) training

vision: true turns the SFT trainer into a VLM fine-tuner on any base whose exllamav3 architecture has a vision component (Qwen2.5-VL, Qwen3-VL, Qwen3.5-VL incl. MoE, Gemma3, Gemma4, Mistral Small 3.x). The approach mirrors how Axolotl does multimodal fine-tuning: the vision tower + projector stay frozen, LoRA targets only the language model, and the dataset is OpenAI-style messages whose content is a parts list:

{"messages": [
  {"role": "user", "content": [
    {"type": "image", "path": "images/0001.png"},
    {"type": "text",  "text": "What breed is this dog?"}]},
  {"role": "assistant", "content": "A border collie."}]}

An image part names its pixels by path (relative to the dataset file), url, base64, image (a PIL image / HF datasets.Image column, e.g. from a parquet set) or OpenAI's image_url; a bare {"type": "image"} part takes the next entry of the row's images column (images_key), the layout of most HF VLM sets. Images can sit anywhere in any turn, several per row, on every prompt format (auto, the explicit formats, or jinja); text-only rows in the same set train as usual.

What happens under the hood, and why it's exact:

• Features come from exllamav3's own vision component (Model.from_config(config, component="vision") → get_image_embeddings), the forward the generator runs. They're encoded once at dataset build (cached in CPU RAM up to vision_cache_gb; if everything fits, the tower is unloaded again to free its VRAM) and spliced into the text tower's embedding stream at the image positions — the same indexed_embeddings substitution modules.Embedding performs at inference. The image token layout is the arch's own (<|vision_start|> + N slots + <|vision_end|> on Qwen-VL, <start_of_image> + 256 + <end_of_image> on Gemma3, [IMG]/[IMG_BREAK] rows on Mistral3, ...), so train and serve see identical token streams. Image tokens are always masked.
• The text tower's multimodal math is reproduced differentiably: Qwen-VL's 3-D mRoPE position ids (a mirror of the gen_mrope_pos_ids kernel, CPU-tested against a transliteration of it) and the per-band interleaved rotation (a mirror of RoPE.get_mrope_freqs); Qwen3-VL / Qwen3.5-VL deepstack (vision intermediate features added after the first blocks — the DeepstackEmbed modules, which the training layout check now accepts instead of rejecting a VL text tower); Gemma4's bidirectional attention inside each image span (the eager attention path takes the span mask on those batches).
• A correctness gate: qlora_validate_native.py --image cat.png runs one image through the vision tower and compares the differentiable forward (features through the training splice) against the native multimodal forward (the generator's prefill params), position by position — plus the Python mRoPE ids against the extension kernel. Run it on a new base before a vision: true run, exactly as for a new architecture.

Knobs: image_max_pixels bounds the image token count (the VRAM lever — Qwen-VL at its default bounds can emit thousands of tokens per image), vision_cache_gb the feature cache, vision_device where the tower loads. --inspect N shows each example's image count / token positions. Not combinable with pack (image rows aren't packed, as in Axolotl); single/split only. Status: the whole path is built and CPU-tested but awaits its first box run (see the handoff log's box checklist).

Modern PEFT: SVD adapter initializations

Short SFT runs spend a large fraction of their steps just growing the adapter off the ground (the default zero-init of B). This fork implements the current crop of SVD-based initializations, adapted to an immutable quantized base — select with one config key, init_lora:

• pissa (PiSSA) — the adapter starts as the top-r principal components of the base weights, trained against a residual base realized as a frozen offset (the trellis itself is never rewritten). Exports as a converted rank-2r standard LoRA so any consumer loads it correctly. Current default recommendation: it won its first A/B clearly (use alpha = r).
• eva (EVA) — A is initialized to the top-r right-singular vectors of each layer's input activations, streamed from your actual training data through the actual quantized forward in a short pre-pass. Function-preserving at step 0. Freshly built; being evaluated against pissa now.
• qerr (LoftQ-style, single-shot) — the adapter starts as the closest rank-r repair of the quantization error vs the original bf16 weights, aimed at the low-bpw regime where that error is large.
• use_rslora — rank-stabilized scaling (alpha/sqrt(r)) for rank sweeps.

Each init has a hard step-0 gate in qlora_validate_native.py --init-lora <mode>.

Credits

The DPO/KTO/SimPO preference-training implementation follows the loss semantics of HuggingFace TRL's stable DPOTrainer, KTOTrainer (KTO stabilized in trl#6175), and CPOTrainer (loss_type="simpo"). TRL is Apache-2.0 licensed, Copyright The HuggingFace Team; this fork reimplements the formulations independently against the EXL3 native training path rather than reusing TRL code. Underlying methods: DPO (Rafailov et al. 2023), KTO (Ethayarajh et al. 2024), IPO, SLiC, SimPO (Meng et al. 2024), CPO (Xu et al. 2024).

Project status

Research project under active development. It started as an exllamav3 fork, but has diverged enough to stand alone (upstream is still merged in periodically — currently at v1.6.1 parity — so the inference side stays current). The core mechanism is proven end-to-end: validated forward parity on the quantized weights, healthy trainings from 1B to 16B models on 1–2× RTX 3090 (including 8k-context packed runs on a 12B), adapters that load and steer generation on the native inference path. Training-side architecture support currently covers Llama-family, Gemma 3/4 (incl. Gemma4 MoE), Qwen3-dense, Qwen3-MoE, Qwen3.5/3.6 hybrids (differentiable Gated DeltaNet + gated attention) incl. Qwen3.5-MoE, AFMoE (Trinity-Nano — dots sigmoid router, gated attention, NoPE), Mistral(-Nemo) dense models, MuseGlimmer text towers (built, awaiting its first box validation), LFM2 / LFM2-MoE hybrids (LFM2.5-8B-A1B — differentiable ShortConv layers; built, awaiting its first box validation), and Spark-X2.5 (headwise sigmoid attention gate + 3:1 sliding/full layers; built, awaiting its first box validation), and Qwen3.8-Flash-Next (125B MoE on a 4-stream hyper-connection residual with a hashed n-gram PLE layer and QSA attention; built, awaiting its first box validation) (see the supported-architectures table above; no sample packing on Gated DeltaNet or ShortConv models); unsupported features are rejected loudly rather than silently mistrained. Vision (image+text) SFT on the Qwen-VL / Gemma3 / Gemma4 / Mistral3 bases is built and CPU-tested (frozen vision tower, differentiable mRoPE / deepstack / bidirectional image spans, qlora_validate_native.py --image gate) and likewise awaits its first box run — see Vision training. Interfaces may still move between sessions — the full engineering log with per-session results and rationale lives in doc/qlora_handoff.md, and experiment-specific tooling is quarantined in training/experiments/.