Qwen3.8-Flash-Next-int2-mixed-AutoRound-24GB-SGLang

[!IMPORTANT] This repository contains Qwen3.8-Flash-Next quantized to 2-bit mixed precision with AutoRound (auto_round:auto_gptq packing), plus the model's n-gram embedding table as a separate fp8 file. It is text-only: the vision tower and the MTP head of the base checkpoint are not included.

It runs on one 24 GB GPU with 32 GB of host memory through SGLang at commit 73a255206f with the patch set from the companion repository HaberstrohSystems/qwen3.8-flash-next-24gb-sglang. The checkpoint does not load in Transformers, vLLM, llama.cpp or unpatched SGLang: none of them has a 2-bit moe_wna16 path or a memory-mapped mode for the n-gram table.

Measured on an RTX PRO 4000 Blackwell (24 GB, sm_120) with 32 GB of host memory: 54 to 57 tokens per second decode at prompts up to 10,001 tokens (50.8 to 52.7 at 41,370 to 257,905 tokens, see Performance), 2,271 tokens per second prefill on a 10,001-token prompt, and a 257,905-token prompt processed end to end with the default KV cache at 262,144 tokens of context.

Highlights

• 2-bit experts, mixed precision. The 512 experts per layer are 2-bit (group size 128, symmetric); the Gated DeltaNet in_proj_qkv/in_proj_z/out_proj, the QSA q/k/v/o_proj and lm_head are 8-bit; routers, norms, embeddings, the QSA indexer and the state parameters stay 16-bit. 2.572 bits per weight for the AutoRound output before the INT8 re-pack.
• Expert placement by routing mass. The 184 most-routed experts of every layer are resident in VRAM (84.3% of the routing mass); the rest is fetched from host memory per token as single 1.33 MB rows.
• Virtualized expert cache. Expert rows sit in CUDA virtual-memory arenas behind an int64 address table; the cache resizes at run time and the captured decode graphs stay valid.
• Lazy KV cache. Address space for 262,144 tokens, backed in 2048-token steps, so the expert cache and the KV cache share the VRAM according to the request.
• Tiered KV cache. INT8 for the last 8192 positions, INT4 for the full context: 7,308 B per token at 262,144 tokens; NLL delta within run-to-run noise on the long-text test (−0.0001 nats per token; noise +0.0002), mean |Δlogprob| 0.074 against a 0.059 noise floor, 5/5 needles at 41,370 and 247,629 tokens.
• Memory-mapped n-gram table. The 51.2 GB n-gram embedding table (ple/ in this repository, SGLANG_QWEN4_PLE_MMAP in SGLang) is read from NVMe with MADV_RANDOM and never loaded into host memory or VRAM.

Model overview

• Developer: Haberstroh Systems
• Release date: 2026-09-03
• Version: 1.0.0
• Input / output: text / text
• Weight quantization: 2-bit mixed precision, AutoRound 0.14.2, auto_round:auto_gptq packing (2.572 bits per weight before the INT8 re-pack)
• Activation quantization: none (16-bit activations)
• KV cache: INT8 ring over an INT4 pool (default); INT8-G64 and INT4-G32 selectable
• Serving stack: SGLang commit 73a255206f plus the patch set of the companion repository
• Tested hardware: RTX PRO 4000 Blackwell (24 GB, sm_120), 32 GB host memory, NVMe
• License: Qwen Community License 1.0

Architecture facts are those of the base model (Qwen/Qwen3.8-Flash-Next-FP8, blog, technical report):

• Type: causal language model (Qwen4ExpForConditionalGeneration, served with --language-model-only)
• Parameters: 125B with 6B activated, plus 51B n-gram embedding (176B total in this repository; the 4B MTP head is not included)
• Hidden dimension: 2560
• Token embedding and LM output: 248,320 (padded)
• N-gram embedding: 20,000,000 (bigrams/trigrams at layer 2)
• Layers: 48, layout 12 × (3 × (Gated DeltaNet → MoE) → 1 × (Qwen Sparse Attention → MoE)), i.e. 36 Gated DeltaNet and 12 QSA layers
• Gated DeltaNet: 48 value heads, 16 QK heads, head dimension 128
• Qwen Sparse Attention: 24 query heads, 2 KV heads, head dimension 256, RoPE dimension 64; indexer MQA with 4 query heads and 1 shared key head, indexer head dimension 128, budget 512 blocks or 2048 tokens
• Mixture of experts: 512 experts, 10 routed + 1 shared per token, expert intermediate dimension 640
• Gated residual: 4 branches, bottleneck rank 320
• Context length: 262,144

What this repository changes: expert and dense weights are quantized as described under Quantization; the n-gram table is shipped as one fp8 file plus its layout (ple/); the vision tower and the MTP head are removed; the tokenizer, chat template, generation configuration and context length are unchanged.

Repository contents

Total download: 38.79 GB of safetensors files (file sizes, including headers) plus 51.20 GB for the n-gram table, 89.99 GB of weight files (about 90.03 GB with the index, tokenizer and configuration files).

Quantization

Quantized from the Qwen FP8 release with AutoRound 0.14.2 on the serving machine (RTX PRO 4000 Blackwell, 32 GB host) in 3 days 8 hours: symmetric 2-bit, group size 128, auto_round:auto_gptq packing, --enable_alg_ext (SignRoundV2), 48 calibration samples of 2048 tokens from AutoRound's default set (NeelNanda/pile-10k), 200 iterations, batch size 1 with gradient accumulation 8, and four subset rounds over the 512 experts of every block. Precision is assigned by sensitivity:

Rationale:

• The routers are never quantized: a rounding error there does not perturb an output, it routes the token to a different expert.
• All expert projections share one (bits, group_size): SGLang's FusedMoE requires a single scheme per layer.
• in_proj_a and in_proj_b are [48, 2560]; AutoRound skips tensors whose 48-wide dimension is not a multiple of 32, so these two stay 16-bit.
• The n-gram table keeps the fp8 of the base release: it is a lookup table, so an error is not averaged over a matmul.
• lm_head and the 84 dense attention projections were re-packed from 16-bit to INT8 (group 128, symmetric) after the AutoRound run; the change is neutral on the held-out texts (see Serving fidelity) and returns about 1.6 GiB of VRAM.

Result: 2.572 bits per weight for the AutoRound output before the INT8 re-pack; this repository ships 38,755,352,600 B of language-model tensors plus the fp8 n-gram table.

--scheme W2A16 --group_size 128 --layer_config "$LAYER_CONFIG" \
--format auto_round --iters 200 --nsamples 48 --seqlen 2048 --batch_size 1 --gradient_accumulate_steps 8 \
--device_map cuda --low_gpu_mem_usage --low_cpu_mem_usage --enable_alg_ext

Requirements

How to serve

Install

# 1. a Python 3.12 virtualenv with torch for CUDA 13 (Triton 3.7.1 comes with it)
mkdir -p ~/quant && cd ~/quant
python3.12 -m venv venv-sglang && . venv-sglang/bin/activate
pip install torch==2.13.0 --index-url https://download.pytorch.org/whl/cu130

# 2. the companion repository and SGLang at the base commit, with the patch set applied
git clone https://github.com/HaberstrohSystems/qwen3.8-flash-next-24gb-sglang
git clone https://github.com/sgl-project/sglang && cd sglang
git checkout 73a255206f916366c8d26d4022f82ddfb0ab558d
git apply --check ../qwen3.8-flash-next-24gb-sglang/sglang/qwen4exp-serving-73a255206f.patch
git apply         ../qwen3.8-flash-next-24gb-sglang/sglang/qwen4exp-serving-73a255206f.patch
pip install -e python

# 3. the CUDA 13 compiler and headers inside the virtualenv (site-packages/nvidia/cu13/), then FlashAttention 2 for sm_120
pip install nvidia-cuda-nvcc==13.3.73 nvidia-cuda-runtime==13.3.29
$VIRTUAL_ENV/lib/python3.12/site-packages/nvidia/cu13/bin/nvcc --version      # release 13.3, V13.3.73
export CUDA_HOME=$VIRTUAL_ENV/lib/python3.12/site-packages/nvidia/cu13
export PATH=$CUDA_HOME/bin:$PATH
FLASH_ATTN_CUDA_ARCHS=120 MAX_JOBS=4 pip install --no-build-isolation --no-deps flash-attn==2.8.3.post1

# 4. the weights (about 90 GB: checkpoint shards plus ple/)
hf download HaberstrohSystems/Qwen3.8-Flash-Next-int2-mixed-AutoRound-24GB-SGLang --local-dir ~/quant/model

SGLang compiles its kernels for compute_120a at run time with the nvcc inside the virtualenv (a system CUDA 12 cannot target that architecture), and tilelang requires the compiler and the runtime headers to share the minor version, hence the two pins. The companion repository ships the two serving assets used below: assets/moe_configs/ (tuned Triton configs for the 2-bit expert GEMM on sm_120) and assets/expert_freq.pt (routing-mass histogram, 48 × 512, used for expert placement).

Launch

Single stream, full 262,144-token context, tiered KV cache. The virtualized expert cache (SGLANG_MOE_ELASTIC*, "elastic" in the companion repository) is controlled through a small text file; S 184 sets 184 resident experts per layer:

VENV=~/quant/venv-sglang                                  # from the install above
CU=$VENV/lib/python3.12/site-packages/nvidia/cu13
MODEL=~/quant/model                                       # the downloaded copy of this repository
PLE=$MODEL/ple
ASSETS=~/quant/qwen3.8-flash-next-24gb-sglang/assets
echo "S 184" > /tmp/elastic.ctl          # control file of the virtualized expert cache

systemd-run --user --scope -p MemoryMax=30G \
  env PATH="$CU/bin:$VENV/bin:$PATH" CUDA_HOME="$CU" \
      SGLANG_QWEN4_PLE_MMAP="$PLE" SGLANG_VLM_CACHE_SIZE_MB=0 SGLANG_MOE_EXPERT_STREAM=1 \
      SGLANG_MOE_CONFIG_DIR="$ASSETS/moe_configs" \
      SGLANG_MOE_PLACEMENT="$ASSETS/expert_freq.pt" SGLANG_MOE_PLACEMENT_S=184 \
      SGLANG_MOE_ELASTIC=1 SGLANG_MOE_ELASTIC_PIN_MB=512 SGLANG_MOE_ELASTIC_CTL=/tmp/elastic.ctl \
      SGLANG_MOE_ELASTIC_FILL_MB=2048 SGLANG_MOE_ELASTIC_RESERVE_ROWS=0 \
      SGLANG_KV_LAZY=1 SGLANG_KV_LAZY_TOKENS=262144 SGLANG_KV_LAZY_SAFETY=0.77 \
      SGLANG_KV_TIERS_W=8192 \
  "$VENV/bin/python3" -m sglang.launch_server \
    --model-path "$MODEL" --host 127.0.0.1 --port 30000 --tp-size 1 \
    --cpu-offload-gb 19 --no-ple-offload-embedding --mem-fraction-static 0.95 \
    --language-model-only --page-size 1 --disable-overlap-schedule --disable-radix-cache \
    --weight-loader-drop-cache-after-load \
    --chunked-prefill-size 1024 --max-prefill-tokens 32768 \
    --cuda-graph-backend-decode breakable \
    --max-total-tokens 262144 --context-length 262144 \
    --kv-cache-dtype int8ring_int4 --attention-backend triton \
    --max-mamba-cache-size 1 --max-running-requests 1 \
    --reasoning-parser qwen3 --tool-call-parser qwen3_coder > server.log 2>&1 &

The systemd-run --scope -p MemoryMax=30G wrapper is part of the configuration: it bounds the server's host memory at 30 GB, and a 32 GB host needs this bound because the offloaded expert rows occupy about 24 GB of it. The server runs in the background and writes its log to server.log; start-up takes about three minutes (weight load, then a 4 s CUDA-graph capture). Wait for /health, then send one warm-up request and freeze the garbage collector:

until curl -sf http://127.0.0.1:30000/health >/dev/null; do sleep 10; done
curl -s http://127.0.0.1:30000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"local","messages":[{"role":"user","content":"Say hello."}],"max_tokens":32}'
curl -s -X POST http://127.0.0.1:30000/freeze_gc

scripts/serve.sh of the companion repository performs exactly this sequence (systemd scope, health wait, warm-up request, POST /freeze_gc) with the same flags and environment.

SGLang serves a single model and echoes the model field of a request back without checking it, so any identifier works; the examples in this card use local.

What the flags do:

• --cpu-offload-gb 19 is a budget for expert weights only. Attention, Gated DeltaNet and norms stay on the GPU; only the routed experts that are not resident cross PCIe.
• SGLANG_MOE_PLACEMENT_S=184: the 184 most-routed experts of every layer are resident (11.06 GiB, 11.88 GB, of expert rows); the other rows live in host memory. S can be changed at run time by writing S <n>, fill [MB] (grow, keeping MB of VRAM free, default 512), free <MB> (shrink until MB of VRAM is free) or status into the control file.
• SGLANG_KV_LAZY_TOKENS=262144: the KV cache is reserved as address space for the full context and backed on demand; SGLANG_KV_LAZY_SAFETY=0.77 caps admitted prompts at 77% of the profiled capacity. A prompt above the cap is rejected at admission with an error response.
• --kv-cache-dtype int8ring_int4: the tiered KV cache (see How it fits into 24 GB). int8_g64 and int4_g32 select the single-precision variants; int8_g64 admits about 161k tokens (0.77 × the 209,600-token profiled capacity of the INT8 pool; the measured 162,215-token run used a 0.85 cap).
• --max-running-requests 1 --max-mamba-cache-size 1: one request at a time, the configuration validated for a 32 GB host.
• --reasoning-parser qwen3 separates the thinking output into reasoning_content; --tool-call-parser qwen3_coder converts the chat template's <tool_call><function=…><parameter=…> XML format into OpenAI tool_calls objects.

Sampling parameters

The base model's recommended settings (Best Practices of the base card) apply unchanged; SGLang supports all of them:

• Thinking mode: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0
• Instruct (non-thinking) mode: temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0

The shipped generation_config.json carries the thinking-mode defaults (temperature 1.0, top_k 20, top_p 0.95, do_sample true), so requests that set no sampling parameters use them.

Client example

The server speaks the OpenAI Chat Completions API on port 30000. Thinking is on by default; enable_thinking, preserve_thinking and reasoning_effort are handled by the shipped chat template exactly as on the base model.

pip install -U openai
export OPENAI_BASE_URL="http://127.0.0.1:30000/v1"
export OPENAI_API_KEY="EMPTY"

from openai import OpenAI
client = OpenAI()   # configured by the environment variables above

MODEL = "local"     # any identifier: SGLang serves one model and does not check the field

completion = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": "Write a Python function to merge two sorted linked lists."}],
    extra_body={
        "chat_template_kwargs": {
            "enable_thinking": True,     # on by default
            "preserve_thinking": True,   # on by default: thinking blocks of earlier turns stay in the context
        },
    },
    reasoning_effort="xhigh",            # xhigh by default; supported levels are xhigh, medium and low
    stream=True,
)

for chunk in completion:
    if not chunk.choices:
        continue
    delta = chunk.choices[0].delta
    if getattr(delta, "reasoning_content", None):
        print(delta.reasoning_content, end="", flush=True)   # thinking, from --reasoning-parser qwen3
    if delta.content:
        print(delta.content, end="", flush=True)             # the answer

Non-thinking mode with the instruct sampling set:

response = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": "Summarize the Qwen Community License in three sentences."}],
    temperature=0.7,
    top_p=0.8,
    presence_penalty=1.5,
    extra_body={"top_k": 20, "chat_template_kwargs": {"enable_thinking": False}},
)
print(response.choices[0].message.content)

Tool calling: pass tools=[...] as usual. The chat template renders the tool schemas, the model answers in its XML function format, and --tool-call-parser qwen3_coder returns parsed tool_calls:

tools = [{"type": "function", "function": {"name": "bash",
          "description": "Run a shell command",
          "parameters": {"type": "object", "properties": {"command": {"type": "string"}},
                         "required": ["command"]}}}]
response = client.chat.completions.create(
    model=MODEL, tools=tools,
    messages=[{"role": "user", "content": "List the files in /tmp."}],
)
print(response.choices[0].message.tool_calls)
# [ChatCompletionMessageToolCall(id='call_…', type='function',
#    function=Function(name='bash', arguments='{"command": "ls -la /tmp"}'))]

How it fits into 24 GB

Expert placement by routing mass. Each token uses 10 of 512 experts per layer, and the routing mass is unevenly distributed. Measured on a routing probe of 2,496 decode tokens over 48 layers and three text domains (assets/expert_freq.pt), the 184 heaviest experts per layer are kept on the GPU (11.06 GiB, 11.88 GB; 84.3% of the routing mass); the rest stay in host memory, about 24 GB of pinned memory. A routed expert that is resident is computed directly; a non-resident one is fetched as a single 1.33 MB row, so the PCIe traffic per token is a few expert rows rather than whole layers.

Virtualized expert cache. Every layer owns an address window from CUDA's virtual memory management. An int64 table maps each expert to the address of its row, in the GPU window or in its host slot, and the decode GEMV and the prefill gather read through that table. Moving an expert is a table write plus one row copy; the captured CUDA graphs stay valid because they only hold the table. Shrinking unmaps 4 MiB chunks (the device granularity is 2 MiB) and returns VRAM to the driver; growing maps them back. A control file resizes the cache while serving.

Lazy KV cache. The KV cache is reserved as address space for 262,144 tokens and backed in 2048-token steps as pages are handed out, then released after the request. During a long prefill the expert cache gives rows back down to its floor under a watermark rule and regrows afterwards, so the two caches share the VRAM according to the request. Prompts that cannot be backed are rejected at admission (SGLANG_KV_LAZY_SAFETY).

Tiered KV cache. The sparse-attention kernels of this model read bf16, so the KV formats are custom: INT8-G64 (one fp16 absmax scale per token, head and 64-channel group; one fused Triton write kernel; dequantization inside the two gather sites; 12,672 B per token), INT4-G32 (nibble-packed, 6,912 B per token) and the default tier: every new token is written as INT8 into a ring of the last 8192 tokens and as INT4 into the full-context pool, and the kernels test on the device whether a position is still in the ring. Recent context keeps 8-bit precision, old context costs 4 bits: 7,308 B per token at 262,144 tokens (6,912 B pool + 8,192 × 12,672 B = 103.8 MB fixed ring).

N-gram table. The 51.2 GB n-gram table (ple/ple.f8_e4m3.bin) is memory-mapped with MADV_RANDOM, and its pages are dropped from the page cache after every bulk gather, so it costs neither host memory nor VRAM beyond the rows currently in use.

Performance

Streaming benchmark (inter-token time over 200 generated tokens), RTX PRO 4000 Blackwell, single stream, default configuration above unless noted in the KV mode column:

Per-chunk prefill time on the same 257,905-token prompt follows 615 ms + 0.487 µs × prefix tokens (fit on a 171.0 s run of this prompt without the rate-limited empty_cache of the serving configuration, which gives the 165.3 s above); the linear term is the sparse-attention indexer of the model. Random-word text (the needle tests) prefills more slowly than prose because every n-gram lookup is a fresh row from NVMe.

The published files were checked after the upload: the SHA-256 of every weight file on the Hub matches the local release copy, and that copy, launched with the command above, answered /health after 164 s and measured 57.1 to 58.2 tokens per second decode and 2,338 tokens per second prefill at 10,001 tokens (2026-09-03).

Serving fidelity

This section measures the serving path (KV cache formats, the INT8 re-pack of the dense projections) against the quantized model's own teacher-forced output. It is not a comparison with the FP8 base model. Four tests:

• Short held-out NLL: teacher-forced negative log-likelihood on three held-out passages of about 700 tokens (German, English, Python; reference values of the quantized model with a bf16 KV cache and 16-bit dense projections: 1.512, 1.278 and 0.414 nats per token). Short prompts never read the KV cache, so this test sees weight-side changes only.
• Logprob oracle: mean absolute log-probability difference per token on fixed continuations after a prompt of about 10k tokens, against saved log-probabilities of the same quantized model; its run-to-run floor is 0.0019.
• All-position long-text NLL: every position from the second prefill chunk on (8,561 positions of a 9,586-token text), i.e. the positions whose attention reads the KV cache. NLL delta is the mean change in nats per token, mean |Δlogprob| the mean absolute log-probability difference at those positions, both against the bf16 KV pool.
• Needle retrieval: five codes hidden at 10, 30, 50, 70 and 90% depth of a random-word haystack; the score is the number retrieved.

INT8-G64 was validated on a 512-position window of the same long text against the bf16 KV pool: mean |Δlogprob| 0.094 (bf16 run-to-run noise on that window 0.099), NLL delta +0.010 (noise ±0.008); its short held-out NLL moves by −0.001 nats per token and its logprob oracle reads 0.0019, the run-to-run floor.

With the default tiered cache the short held-out NLL moves by −0.0002 nats per token and the logprob oracle reads 0.0019, the run-to-run floor. The INT8 re-pack of lm_head and the dense attention projections changes the short held-out NLL by −0.016 (German), +0.010 (English) and +0.005 (Python), −0.001 nats per token overall.

Scope and limitations

• Supported: text generation, chat, reasoning (thinking mode) and tool calling. The vision tower is not included, so image and video inputs are not supported.
• One request at a time on a 32 GB host. More concurrency needs more host memory: the pinned expert rows already occupy about 24 GB, and parallel requests multiply the row traffic.
• The MTP head of the base checkpoint is not included; speculative decoding is not part of the serving configuration.
• 262,144-token context with the tiered or INT4 KV cache; about 161k tokens with INT8-G64 (0.77 × the 209,600-token profiled capacity of the INT8 pool; the measured 162,215-token run used a 0.85 cap).
• The tuned Triton configurations and the measurements are specific to the RTX PRO 4000 Blackwell; other GPUs need their own tuning run.
• The usage terms and the behavior of the base model apply unchanged.

License and acknowledgements

The weights are released under the Qwen Community License 1.0 (LICENSE), inherited from the base model Qwen/Qwen3.8-Flash-Next-FP8 by the Qwen team. Quantization with AutoRound (Intel). Serving with SGLang. The patch set, kernels, tools and the complete dated engineering log are in the companion repository HaberstrohSystems/qwen3.8-flash-next-24gb-sglang (Apache-2.0).

Citation

This repository redistributes quantized weights. Please cite the base model and, if you use this artifact, the repository itself:

@techreport{qwen2026design,
    title       = {On the Design of {Qwen3.8-Next} Architecture: Evaluation, Efficiency, and Training Stability},
    author      = {{Qwen Team}},
    institution = {Alibaba Group},
    month       = {August},
    year        = {2026}
}

@misc{qwen3.8flashnext,
    title  = {{Qwen3.8-Flash-Next}: A New Architecture, Towards Ultimate Cost-Efficiency},
    author = {{Qwen Team}},
    month  = {August},
    year   = {2026},
    url    = {https://qwen.ai/blog?id=qwen3.8-flash-next}
}

@misc{haberstroh2026qwen38flashnext24gb,
    title        = {{Qwen3.8-Flash-Next-int2-mixed-AutoRound-24GB-SGLang}: 2-bit mixed-precision Qwen3.8-Flash-Next on one 24 GB GPU},
    author       = {Haberstroh, Maximilian Roland},
    howpublished = {Hugging Face model repository, Haberstroh Systems},
    year         = {2026},
    url          = {https://huggingface.co/HaberstrohSystems/Qwen3.8-Flash-Next-int2-mixed-AutoRound-24GB-SGLang}
}