Qwen3.8-27B-NVIDIA-NVFP4-NInferV3

A single-file NInfer V3 (.ninfer) artifact of nvidia/Qwen3.8-27B-NVFP4, built for an NVIDIA RTX 5090 (sm_120a), for Infernix (recommended) and NInfer. No conversion step at load time: point infernix-serve (or ninfer-serve) at the .ninfer file and run.

This repository is the NInfer form of the NVIDIA checkpoint above: the Model Optimizer mixed NVFP4/FP8 weights are carried over bit-exactly for every transformer linear except the language-model head, and two speculative-decoding components (DFlash2 draft model, indexed proposal head) are added so the artifact runs with the engines' fastest decode paths.

Recommended engine: Infernix, my engine grown from NInfer, which replaces my earlier ninfer-custom fork. In an agentic coding replay on the 27B it served 90 % of prompt tokens from its prefix cache and reached the first token in about half NInfer's average time, and it prefills long prompts 1.3-1.5× faster, at the same quality (benchmarks, measured on NInfer's official 27B artifact). It also runs Qwen3.8-Flash-Next. The artifact still works with NInfer itself and with ninfer-custom.

What is in the file

Format census of the 1,072 prepared weight objects: 128 NVFP4 (MLP projections, imported bit-exact), 130 FP8 (e4m3fn, row-scale; attention + GDN projections imported bit-exact, plus the re-quantised embedding and output head), 579 BF16, 96 FP32, 55 Q4, 54 Q5, 28 Q8 and 1 Q6 (grouped, FP16 scales), and 1 INT32 proposal index table.

How it was created

The artifact was built with the NInfer V3 converter using the qwen3_8_27b_nvfp4_nvidia recipe (tools/convert/official_recipes.py) on CPU (~28 min). qwen3_8_27b_nvfp4-nvidia.ninfer.conversion.json, included here, is the full per-tensor conversion report. Sources and treatment:

1. nvidia/Qwen3.8-27B-NVFP4 (the base model, Model Optimizer mixed NVFP4/FP8 in compressed-tensors format) — the dominant source.
   • MLP projections (gate/up/down, all 128 NVFP4 objects) are imported in their encoded NVFP4 form, bit-exact (import_encoded): the Local-Hessian-calibrated packed FP4 weights and their block scales are carried over unchanged, with no re-quantisation round-trip. At runtime these drive NInfer's W4A4 NVFP4 kernels, which dequantise weights on the fly against the stored scales and quantise activations in-kernel.
   • Self-attention and linear-attention (GDN) projections (128 FP8 objects) are likewise imported in their encoded per-row FP8 form, bit-exact (import_encoded): the source's E4M3 weights and row scales are carried over unchanged.
   • Output head (text/output_head) — the source stores this as NVFP4, but NInfer registers the vocabulary projection only for Q8/Q6/FP8, so it is dequantised and re-quantised to row-scale FP8 (fp8_row_maxabs). FP8 was chosen over Q8 after benchmarking the head op: it is faster across the decode/verify token range the engine uses (T=1..33); Q8 wins only at T=34..64.
   • Token embedding is quantised from its source BF16 to row-scale FP8.
   • GDN a/b projections stay in their source BF16: the v3 NVFP4 storage layout requires 128-row tiles and these are 96-row tensors, so they are kept unquantised (the same values, stored in a wider format). Norms, convolutions, dt_bias/a_log and the other non-projection tensors keep their source BF16/FP32 formats.
   • Vision tower and MTP companion are carried over from the checkpoint (the vision tower in its mixed BF16 + grouped Q4/Q5/Q6/Q8 storage; the MTP companion in BF16 + Q8).
2. DFlash2 draft — the 5-layer sliding-attention draft model is grafted verbatim from a local DFlash2 source (the same draft used in the Quasar artifact) so the --spec dflash2 path runs on this backbone.
3. Indexed proposal head — 131,072 rows gathered from the NVIDIA output head, ranked by a frequency corpus, stored as grouped Q4 plus an INT32 token-id table; this is what --lm-head-draft serves copy-heavy proposals from.
4. Resources — the six tokenizer / chat-template / generation-config resources are byte-identical to the official NInfer artifact. The NVIDIA checkpoint's own tokenizer assets omit the added_tokens_decoder and standard Split pattern the C++ tokenizer hard-requires, so they are normalised at conversion time.

No MLP or attention weight is dequantised and then re-quantised: the 128 NVFP4 MLP and 128 FP8 attention/GDN objects are direct imports of the checkpoint's stored weights, so the base model's quality is preserved by construction. The only re-quantisations are the output head (NVFP4 → FP8, an engine op requirement) and the token embedding (BF16 → FP8).

How to run

Requires an NVIDIA RTX 50-series GPU (built and tuned on an RTX 5090; the weights take ~21 GB of VRAM). Prebuilt Windows executables of Infernix are on its releases page and need an NVIDIA driver of 580 or later; on Linux, build from source (quick start).

infernix-serve.exe qwen3_8_27b_nvfp4-nvidia.ninfer --host 127.0.0.1 --port 8080 --max-context 240000 --max-concurrency 2 --spec dflash2 --draft-tokens 7 --lm-head-draft --ngram-draft-tokens 15 --ngram-min-match 12 --kv-dtype int8 --preserve-thinking --host-context-mib 52000 --pending-timeout-ms 900000 --prefill-chunk 4096 --kv-capacity auto --vram-headroom-mib 0 --log-colours on --ngram-archive-mib 2048 --ngram-session-mib 256 --ngram-native-sessions --request-log-jsonl log.json --default-thinking-budget 16384 --thinking-budget-message "Considering the limited time available to the user, I must stop thinking now. Time to act:" --tolerant-tool-calls

It runs DFlash2 speculative decoding with 7 draft tokens, the indexed proposal head (--lm-head-draft) and n-gram copy drafting, int8 KV, and the hybrid prefix cache with a 52,000 MiB tier in system RAM (--host-context-mib). --kv-capacity auto sizes the KV pool to the free VRAM. Lower --host-context-mib on a machine with less system RAM, and on a single 32 GB GPU stop any other resident model first. Add --prefix-cache-file PATH to keep the prefix cache across restarts. infernix-serve.exe --help lists every option.

With NInfer: the artifact also runs on NInfer and on ninfer-custom with ninfer-serve and the same command; drop any option that engine's ninfer-serve --help does not list.

Tested on 2026-10-09 on an RTX 5090 (Windows) with the Infernix windows-2026.10.09 release: the command above (on another port) started with this artifact and served a chat request, a streamed request, a tool call, and Anthropic Messages and Responses requests.

• Images and video: add --vision. --vision-offload on keeps the vision tower in system RAM instead of VRAM, which frees about 0.8 GiB of VRAM for the KV cache.
• Longer context: beyond the native 262,144 tokens with --rope-yarn-factor F (F from 1 to 4, up to 1M tokens).

---

Original model card (copied verbatim from nvidia/Qwen3.8-27B-NVFP4)

The following is the README of the source checkpoint nvidia/Qwen3.8-27B-NVFP4, copied from its source repository for reference.

---

pipeline_tag: text-generation base_model:

• Qwen/Qwen3.8-27B license: apache-2.0 library_name: Model Optimizer tags:
• nvidia
• ModelOpt
• Qwen3.8
• quantized
• FP4
• fp4
• FP8
• fp8

---

Model Overview

Description:

The NVIDIA Qwen3.8-27B NVFP4 model is a quantized version of Alibaba's Qwen3.8-27B model, which is an auto-regressive language model that uses an optimized transformer architecture. For more information on the model, please check here. The model is quantized with Model Optimizer.

This model is ready for commercial or non-commercial use.

Third-Party Community Consideration

This model is not owned or developed by NVIDIA. This model has been developed and built to a third-party's requirements for this application and use case; see link to Non-NVIDIA (Qwen3.8-27B) Model Card from Qwen.

License/Terms of Use:

GOVERNING DOWNLOAD TERMS: Use of the model is governed by the Apache 2.0

Deployment Geography:

Global

Use Case:

Developers looking to take off-the-shelf, pre-quantized models for deployment in AI Agent systems, chatbots, RAG systems, and other AI-powered applications.

Release Date:

Hugging Face 09/08/2026 via https://huggingface.co/nvidia/Qwen3.8-27B-NVFP4

References

NVIDIA Model Optimizer: https://github.com/NVIDIA/Model-Optimizer

Model Architecture:

Architecture Type: Transformer
Network Architecture: Qwen3.8-27B (Qwen3_5ForConditionalGeneration)
Number of Model Parameters: 27B

Input:

Input Type(s): Text, Image, Video
Input Format(s): String, Red, Green, Blue (RGB), Video (MP4/WebM)
Input Parameters: One-Dimensional (1D), Two-Dimensional (2D), Three-Dimensional (3D)
Other Properties Related to Input: Context length up to 262K

Output:

Output Type(s): Text
Output Format: String
Output Parameters: One-Dimensional (1D): Sequences
Other Properties Related to Output: None

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration:

Supported Runtime Engine(s):

• vLLM
  
• SGLang
  

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Blackwell
  

Preferred Operating System(s):

• Linux
  

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Version(s):

This checkpoint uses mixed NVFP4/FP8 quantization and was produced with nvidia-modelopt v0.48.0.

Training and Evaluation Datasets:

Calibration Dataset:

Link: Nemotron-Post-Training-Dataset-v3
Data Modality: Text
Data Collection Method by dataset: Varies by dataset.
Labeling Method by dataset: Varies by dataset.
Properties: The Nemotron-Post-Training-Dataset-v3 is a multi-million-sample corpus developed by NVIDIA for Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to power alignment, reasoning, and agentic capabilities in the Nemotron-3 model family.

Training Dataset:

Data Modality: Undisclosed
Data Collection Method by dataset: Undisclosed
Labeling Method by dataset: Undisclosed
Properties: Undisclosed
Image Training Data Size: Undisclosed
Text Training Data Size: Undisclosed
Video Training Data Size: Undisclosed

Evaluation Dataset:

Datasets: GPQA Diamond, Terminal-Bench, AA-LCR, MMMU-Pro, SciCode, IFBench
Data Collection Method by dataset: Hybrid: Automated, Manually-Collected
Labeling Method by dataset: Hybrid: Manually-Labeled, Automated
Properties: We evaluated the model on text-based reasoning, coding, agentic tasks, long-context recall, instruction following, and multimodal reasoning. GPQA Diamond contains graduate-level multiple-choice questions written by domain experts in biology, physics, and chemistry. Terminal-Bench evaluates agents on terminal-based tasks. AA-LCR (Artificial Analysis Long Context Recall) evaluates a model's ability to accurately retrieve and recall information from long input contexts. MMMU-Pro is a challenging multimodal understanding benchmark that measures college-level reasoning across diverse disciplines. SciCode evaluates scientific coding capabilities. IFBench evaluates instruction-following capabilities across diverse and structured task constraints.

Inference:

Acceleration Engine: vLLM
Test Hardware: NVIDIA Grace Blackwell GB300

Post-Training Quantization

Qwen3.8-27B was quantized using a mixed-precision recipe. NVFP4 quantization was applied to the MLP layers and language model head (lm_head), while FP8 quantization was applied to the self-attention and linear-attention layers. The NVFP4 layers were calibrated on 2,048 samples using the Model Optimizer Local-Hessian calibration algorithm.

To learn more about the algorithm, see Local Hessian for NVFP4 Quantization. To reproduce this checkpoint, follow the command in the Using Local Hessian section.

Usage

To serve this checkpoint with vLLM, you can start the docker vllm/vllm-openai:nightly and run the sample command below:

vllm serve nvidia/Qwen3.8-27B-NVFP4 \
    --port 8000 \
    --kv-cache-dtype fp8_e4m3 \
    --tensor-parallel-size 4 \
    --max-model-len 262144 \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --mm-encoder-tp-mode data \
    --seed 0 \
    --gpu-memory-utilization 0.85 \
    --max-num-seqs 32 \
    --max-num-batched-tokens 32768 \
    --enable-chunked-prefill

To serve this checkpoint with SGLang, you can start the docker lmsysorg/sglang:dev and run the sample command below:

sglang serve \
  --trust-remote-code \
  --model-path nvidia/Qwen3.8-27B-NVFP4 \
  --kv-cache-dtype fp8_e4m3 \
  --mem-fraction-static 0.85 \
  --chunked-prefill-size 2048 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --mamba-full-memory-ratio 4.59 \
  --host 0.0.0.0 \
  --port 30000 \
  --mamba-radix-cache-strategy extra_buffer \
  --mamba-ssm-dtype float32

For more details please refer to SGLang cookbook.

Evaluation

The accuracy benchmark results are presented in the table below:

Evaluation settings: All evaluations used temperature=1.0, top_p=0.95, and a vLLM context limit of 262,144 tokens. All benchmarks except Terminal-Bench 2.1 used max_new_tokens=65536. Terminal-Bench 2.1 did not set an explicit generation-token limit;

Model Limitations:

The base model was trained on data that contains toxic language and societal biases originally crawled from the internet. Therefore, the model may amplify those biases and return toxic responses especially when prompted with toxic prompts. The model may generate answers that may be inaccurate, omit key information, or include irrelevant or redundant text producing socially unacceptable or undesirable text, even if the prompt itself does not include anything explicitly offensive.

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the video generated will not blur or maintain proportions of image subjects included.

Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns link.