E4B on Orin Nano

This fork adds Gemma 4 E4B text inference and conversion on Jetson Orin to upstream ExLlamaV3. It uses the existing model, paged-cache, dynamic Generator and conversion APIs, including cache-only prefill through the last KV-producing layer. The full target vocabulary is retained.

• Orin setup, conversion and scope
• E4B QAT EXL3 4.0 bpw weights
• little-gemma v1.0 GGUF baseline

Build this fork's source for SM 8.7. The upstream installation instructions and architecture list below describe the base project; upstream wheels do not include this fork's E4B additions. E4B image/audio and tensor-parallel paths remain unvalidated. See the model card for requantization quality limits.

Measured on Orin Nano 8 GB Super (cortex1), MAXN_SUPER with pinned clocks, on 2026-10-11. Median of three warm runs after one discarded run. All engines receive identical input IDs and generate 128 plain greedy tokens; MTP/speculation is disabled. Values are model-only, excluding load and tokenization.

Prefill benefits from the optional FP16 weight cache. At 930 input tokens, the 1536 MiB budget reaches 1,136 tok/s versus little-gemma’s 948, with first token time 878 ms versus 1,018 ms. At 400 tokens, little-gemma still reaches the first token sooner (485 versus 503 ms). little-gemma also retains faster plain decoding. These measurements do not establish speech/first-PCM latency.

The cache is off by default. The 512/1536 MiB budgets actually allocate 507.5/1507.5 MiB and leave minimum whole-system available RAM of 2.15/1.17 GiB in these runs. They keep packed decode weights and add FP16 prefill copies. See the cache API and exact selection policy. Original-basis caching changes rounding: the 930-token generated sequence matched uncached EXL3; the 400-token run matched the first pick and 111 further decode picks before diverging. Last-prompt-logit relative RMSE remained below 0.00274 across these checks. Evaluate this rounding change on your workload.

Installation · Supported models · Examples · Quantization · Community

ExLlamaV3 is an inference library for running local LLMs on modern consumer GPUs, with flexible quantization and parallel inference.

• Quantization - EXL3, based on QTIP, plus 2–8 bit cache quantization.
• Parallel inference - Flexible tensor-parallel and expert-parallel inference for consumer hardware setups.
• CPU offloading - Allows large MoE models to run with limited GPU resources. AVX2 and AVX512 support.
• Generation - Continuous, dynamic batching, speculative decoding, multimodal support.
• Integrations - Broad HF model support, a Transformers plugin, and an OpenAI-compatible API via TabbyAPI.

[!TIP] Looking for a server? TabbyAPI is the official and recommended backend server. It provides an OpenAI-compatible API for local or remote inference, HF model downloading, embedding model support, and HF Jinja2 chat templates. Its startup script manages and installs prerequisites to help you get started.

Installation

Start by making sure you have the appropriate version of PyTorch installed (CUDA 12.4 or later) since the Torch dependency is not automatically handled by pip. Then pick a method below:

Prebuilt wheel · recommended

Pick a wheel from the releases page, then e.g.:

pip install https://github.com/turboderp-org/exllamav3/releases/download/v0.0.6/exllamav3-0.0.6+cu128.torch2.8.0-cp313-cp313-linux_x86_64.whl

Install from PyPI

pip install exllamav3

Note that the PyPI package does not contain a prebuilt extension and requires the CUDA toolkit and build prerequisites (i.e. VS Build Tools on Windows, gcc on Linux, python-dev headers etc.).

Build from source

git clone https://github.com/turboderp-org/exllamav3
cd exllamav3
# (Optional) switch to dev branch for latest in-progress features
git checkout dev

uv venv
uv sync --extra cu130
# add --extra examples and/or --extra eval for those extra dependencies

# `uv add` works inside an existing project (a directory with a pyproject.toml).
# `uv init` creates one if you're starting a new project, if integrating into
# an existing project skip `uv init`.
uv init my-project
cd my-project

# local checkout
uv add 'path/to/exllamav3[cu130]'               # non-editable
uv add 'path/to/exllamav3[cu130]' --editable    # editable

# straight from GitHub
uv add 'git+https://github.com/turboderp-org/exllamav3.git[cu130]'                 # default branch
uv add 'git+https://github.com/turboderp-org/exllamav3.git[cu130]' --branch dev    # specific branch

uv venv            # or: uv venv --python-preference only-managed
source .venv/bin/activate
uv pip install torch --torch-backend=auto
uv pip install .

# install a CUDA-enabled torch first so it matches your setup, e.g.:
pip install torch --index-url https://download.pytorch.org/whl/cu128
pip install .

my-exllamav3-dev/          # thin project (uv init)
├── pyproject.toml
└── src/                  # package sources (auto-generated)

[project]
name = "my-exllamav3-dev"
version = "0.1.0"
description = "Dev environment for exllamav3"
requires-python = ">=3.10.11"
dependencies = [
    "exllamav3[cu130]",   # select correct CUDA version
    "torch==2.13.0",      # pin the exact torch version you need
]

[tool.uv.sources]
exllamav3 = { path = "../exllamav3", editable = true }

After installing with one of the options above, you should be able to run the conversion, eval and example scripts from the main repo directory, e.g., uv run python convert.py -i ... or, for manual installations once the venv is active, python convert.py -i ...

Build environment variables

• MAX_JOBS: by default ninja may launch too many processes and run out of system memory for compilation. Set this to a reasonable value like 4 in that case.
• EXLLAMA_NOCOMPILE: set to install the library without compiling the C++/CUDA extension. Torch will build/load it at runtime instead.
• EXLLAMA_EXT_LINEINFO, EXLLAMA_EXT_COMPRESS: see doc/env_vars.md.

Examples

A number of example scripts are provided to showcase the features of the backend and generator. For instance, a versatile CLI chatbot:

python examples/chat.py -m <input_dir> -mode <prompt_mode>

# Wealth of options
python examples/chat.py -h

Architecture support

Always adding more, stay tuned.

Conversion

To convert a model to EXL3 format, use:

# Convert model
python convert.py -i <input_dir> -o <output_dir> -w <working_dir> -b <bitrate>

# Resume an interrupted quant job
python convert.py -w <working_dir> -r

# More options
python convert.py -h

The working directory is temporary storage for state checkpoints and for storing quantized tensors until the converted model can be compiled. It should have enough free space to store an entire copy of the output model.

See the conversion guide for more information, or the self-calibration guide.

EXL3 quantization

EXL3 quantization is a streamlined variant of QTIP from Cornell RelaxML. It aims to make SOTA quantization available to users on consumer hardware. The conversion process is designed to be simple and efficient and requires only an input model (in HF format) and a target bitrate. By computing Hessians on the fly and thanks to a fused Viterbi kernel, the quantizer can convert a model in a single step, taking a couple of minutes for smaller models, up to a few hours for larger ones (70B+) on a single high-end consumer GPU (see the conversion guide).

For more information, see the QTIP and QuIP# papers, as well as this excellent writeup on QTIP from together.ai.

Community

You are always welcome to join the ExLlama discord server ←🎮

🤗 Models on Hugging Face

Browse the EXL3 model collection for quantized models. Also shout out to the following lovely people:

• ArtusDev
• MikeRoz
• MetaphoricalCode
• Ready.Art
• isogen

Acknowledgements

This project owes its existence to a wonderful community of FOSS developers and some very generous supporters (🐈❤️!) The following projects in particular deserve a special mention:

• TabbyAPI
• PyTorch
• FlashAttention
• QTIP
• Transformers
• Marlin
• Flash Linear Attention (chunked linear-attention prefill kernels, vendored under exllamav3/vendor/fla)