A single-file NInfer V3 (.ninfer) artifact of
orcarouter/Qwen3.8-27B-Uncensored-NVFP4,
built for an NVIDIA RTX 5090 (sm_120a), for Infernix
(recommended) and NInfer. No conversion step at load time:
point infernix-serve (or ninfer-serve) at the .ninfer file and run.
This repository is the NInfer form of the orcarouter checkpoint above, an abliterated (refusal-removed) mixed NVFP4/FP8 build of Qwen3.8-27B. Its NVFP4 and FP8 weights are carried over bit-exactly for every transformer linear, and two speculative-decoding components (DFlash2 draft model, indexed proposal head) are added so the artifact runs with the engines' fastest decode paths.
Recommended engine: Infernix, my engine grown from NInfer, which replaces my earlier ninfer-custom fork. In an agentic coding replay on the 27B it served 90 % of prompt tokens from its prefix cache and reached the first token in about half NInfer's average time, and it prefills long prompts 1.3-1.5Γ faster, at the same quality (benchmarks, measured on NInfer's official 27B artifact). It also runs Qwen3.8-Flash-Next. The artifact still works with NInfer itself and with ninfer-custom.
Disclaimer (inherited from the source model). The orcarouter checkpoint has had its safety alignment substantially removed by abliteration, so it will comply with requests the original Qwen3.8-27B would refuse. The source card releases it strictly for legitimate research (interpretability, AI-safety and refusal-mechanism study, red-teaming, robustness evaluation) and says you assume full responsibility for how you use it and what it generates; add your own safety and moderation layers before exposing it to other users. The abliteration was done by orcarouter, not by me; this repository only changes the storage format. The full source card, including its disclaimer, is reproduced at the bottom.
| Component | Contents |
|---|---|
text | Qwen3.5-family hybrid backbone: 64 layers (Gated Delta-Net linear attention + full attention every 4th layer), hidden 5120, vocab 248,320, native context 262,144 |
vision | qwen3_5_vision tower (depth 27) for image/video input |
mtp | Qwen3.5 MTP companion (the abliterated head from the source) for MTP speculative decoding |
dflash2 | 5-layer sliding-attention draft model (window 2048, selector rank 256 / top-16) β the --spec dflash2 path |
proposal | Indexed 131,072-row proposal head gathered from the source output head β enables cheap copy/proposal drafting via --lm-head-draft |
Format census of the 1,072 prepared weight objects: 112 NVFP4 (MLP projections of layers 0β55, imported bit-exact), 146 FP8 (e4m3fn, row-scale; 144 attention, GDN and last-eight-layer MLP objects imported bit-exact, plus the re-quantised embedding and output head), 579 BF16, 96 FP32, 55 Q4, 54 Q5, 28 Q8 and 1 Q6 (grouped, FP16 scales), and 1 INT32 proposal index table.
The artifact was built with the NInfer V3 converter using the qwen3_8_27b_nvfp4_orcarouter recipe
(tools/convert/official_recipes.py)
on CPU (~12 min), with components text,vision,mtp,dflash2 and --proposal --proposal-rows 131072.
qwen3_8_27b_nvfp4-uncensored.ninfer.conversion.json, included here, is the full per-tensor
conversion report. Sources and treatment:
llmcompressor
compressed-tensors mixed-precision checkpoint, per its model card) β the dominant source.
import_encoded): the packed FP4 weights and their block scales are carried
over unchanged, with no re-quantisation round-trip. At runtime these drive NInfer's W4A4 NVFP4
kernels, which dequantise weights on the fly against the stored scales and quantise
activations in-kernel.fp8_row_maxabs), the only weights of the text model that are re-quantised. The conversion
check measured the maximum relative dequantisation error against the BF16 source at 0.034 for
the embedding and 0.023 for the output head (bound: 0.07, the FP8 E4M3 half-ULP).dt_bias/a_log and the other non-projection tensors keep their source BF16/FP32 formats.--spec dflash2 path runs
on this backbone. It was not retrained for the abliterated weights.--lm-head-draft serves copy-heavy proposals from.generation_config.json and the
preprocessor configs carry the same values as the source).Verification. The converter's own checks passed for every quantisation family, and an exhaustive replay of the deterministic conversion pipeline from the source files (2026-09-27) byte-compared all 1,072 weight objects plus the auxiliary and resource objects against the artifact: zero differences. Beyond that I did not measure quality (no perplexity or benchmark numbers for this artifact), and the source card also lists the quantised checkpoint's capability retention as not yet measured.
Requires an NVIDIA RTX 50-series GPU (built and tuned on an RTX 5090; the weights take ~21.4 GiB of VRAM). Prebuilt Windows executables of Infernix are on its releases page and need an NVIDIA driver of 580 or later; on Linux, build from source (quick start).
set INFERNIX_CUDA_SYNC=blocking
infernix-serve.exe qwen3_8_27b_nvfp4-uncensored.ninfer --host 127.0.0.1 --port 8080 --max-context 210000 --max-concurrency 2 --spec dflash2 --draft-tokens 7 --lm-head-draft --ngram-draft-tokens 15 --ngram-min-match 12 --kv-dtype int8 --preserve-thinking --host-context-mib 52000 --pending-timeout-ms 900000 --prefill-chunk 4096 --kv-capacity auto --vram-headroom-mib 0 --log-colours on --ngram-archive-mib 4096 --ngram-session-mib 256 --ngram-native-sessions --request-log-jsonl log.json --default-thinking-budget 16384 --thinking-budget-message "Considering the limited time available to the user, I must stop thinking now. Time to act:" --tolerant-tool-calls --prefix-cache-file prefixes-uncensored.cache --vision --vision-offload on
On a single 32 GB GPU stop any other resident model first.
The command runs DFlash2 speculative decoding with 7 draft tokens, the indexed proposal head
(--lm-head-draft) and n-gram copy drafting, int8 KV, the hybrid prefix cache with a
52,000 MiB tier in system RAM (--host-context-mib) that is saved to and restored from
--prefix-cache-file across restarts, a thinking budget of 16,384 tokens, and recovery of
slightly malformed tool calls (--tolerant-tool-calls). --vision enables image and video input,
and --vision-offload on keeps the vision tower in system RAM instead of VRAM, which frees about
0.8 GiB of VRAM for the KV cache. INFERNIX_CUDA_SYNC=blocking stops the server keeping one CPU
core at 100% while it runs, at a cost of about 1% in output tokens per second (measured on
ninfer-custom). infernix-serve.exe --help lists every option.
With NInfer: the artifact also runs on NInfer and on
ninfer-custom with ninfer-serve and the same
command; drop any option that engine's ninfer-serve --help does not list, and use
NINFER_CUDA_SYNC in place of INFERNIX_CUDA_SYNC.
Tested on 2026-10-09 on an RTX 5090 (Windows) with the Infernix windows-2026.10.09 release:
the command above (on another port) started with this artifact and served a chat request, a
streamed request, a tool call, Anthropic Messages and Responses requests, and an image request
(which answered "red and blue" for a red image with a blue stripe). An earlier test with
ninfer-custom (2026-10-01) decoded at about 265β295 tokens per second in a handful of requests, with
DFlash2 accepting about 45β53% of drafted tokens. These are smoke tests, not benchmarks; I have not
measured speed or quality against the other artifacts.
The following is the README of the source checkpoint
orcarouter/Qwen3.8-27B-Uncensored-NVFP4, copied from its source repository for reference.
license: apache-2.0 base_model: Qwen/Qwen3.8-27B base_model_relation: quantized pipeline_tag: image-text-to-text library_name: transformers language:
An abliterated (refusal-removed) & dynamic mixed-precision NVFP4 + FP8 build of Qwen's Qwen3.8-27B β for Blackwell FP4
One Gateway. Every Model. β Route Smarter Β· Ship Safer Β· Spend Less.
Run via API Β· API Endpoint Β· Website Β· Model Catalog Β· Model Card Β· GitHub Β· Discord Β· X
An abliterated (refusal-removed) and dynamic mixed-precision NVFP4 + FP8 quantized build of
Qwen/Qwen3.8-27Bβ a 27B-parameter dense, hybrid-attention (Gated DeltaNet linear + full attention) native vision-language model with flexible thinking control, tool-calling, and an MTP speculative-decoding head. This build removes the safety refusal direction, then quantizes the bulk feed-forward layers to 4-bit NVFP4 while keeping precision-sensitive layers and the KV cache at FP8, so accuracy is better preserved than uniform W4A4. 262K context, tools + reasoning + MTP + vision preserved. Browse all models in the OrcaRouter Model Catalog.Sibling releases: Β β’Β
Qwen3.8-27B-Uncensoredβ BF16 source Β β’ΒQwen3.8-27B-Uncensored-FP8β block-FP8 for vLLM Β β’ΒQwen3.8-27B-Uncensored-GGUFβ 2-bitβ16-bit for llama.cpp Β β’ΒQwen3.8-27B-Uncensored-MLXβ MLX for Apple Silicon (2 / 4 / 8-bit).
This model has had its safety alignment substantially removed via abliteration (orthogonalizing the refusal direction out of the residual stream). As a direct consequence:
Qwen3.8-27B would refuse. It has no meaningful built-in guardrails.By downloading or using this model you acknowledge and accept the above.
| Base model | Qwen/Qwen3.8-27B |
| Architecture | Qwen3_5ForConditionalGeneration β 64 layers, hidden 5120, hybrid Gated DeltaNet (48 linear-attention + 16 full-attention, interval 4), native VL tower + MTP head |
| Modification | Abliteration (refusal-direction removal) then dynamic mixed-precision NVFP4 + FP8 quantization |
| Quantization | Mixed-precision (compressed-tensors) β NVFP4 (W4A4) on the FFN of layers 0β55, FP8 (W8A8 dynamic) on attention / GDN projections / the last 8 layers' FFN / lm_head, and a static FP8 KV cache |
| Format | safetensors, resharded to β€ 5 GB shards (5 + 1 shards, 23.4 GB, 1968 tensors) |
| Precision | FP4 E2M1 group-16 (+ FP8-E4M3 block scale + FP32 global scale) and FP8 E4M3; vision tower / norms / GDN in_proj_a/b / embeddings / MTP head kept in BF16 |
| Preserved | Full vision-language tower and MTP speculative-decoding head (drop-in for the base) |
| Context | 262,144 tokens |
Refusal-direction removal following Arditi et al. (2024), Refusal in Language Models Is
Mediated by a Single Direction. A single refusal direction r (k = 1) is estimated as the
massive-activationβmasked mean-difference of harmful β harmless last-token residuals at
layer 38 (round(0.6 Γ 64)), on AdvBench (harmful) vs Alpaca (harmless). r is then
orthogonalized out of every residual-writing matrix β W' = W β r(rα΅W) β computed in
float32:
| Component | matrices edited |
|---|---|
self_attn.o_proj (16 full-attention layers + MTP) | 17 |
linear_attn.out_proj (48 linear-attention / GDN layers) | 48 |
mlp.down_proj (64 layers + MTP) | 65 |
embed_tokens (row space) | 1 |
| Total | 131 |
The vision tower is untouched and the MTP head is abliterated consistently with the
main model, so speculative decoding keeps working. This is the same abliterated BF16 base as the
FP8,
GGUF and
BF16 releases β only the
quantization differs.
Rather than quantizing every linear uniformly, precision-sensitive layers are kept at FP8 while only the bulk feed-forward layers go to 4-bit:
| Component | Precision | # linears |
|---|---|---|
MLP gate/up/down, layers 0β55 | NVFP4 β FP4 E2M1, group 16, FP8-E4M3 block scale + FP32 global scale (W4A4) | 168 |
self_attn.{q,k,v,o}_proj, GDN in_proj_qkv/in_proj_z/out_proj | FP8 β W8A8, per-channel weight, dynamic per-token activation | 208 |
MLP gate/up/down of the last 8 layers (56β63), lm_head | FP8 (W8A8 dynamic) | 25 |
| KV cache | FP8 β static, per-tensor | β |
Vision tower, GDN in_proj_a/in_proj_b, all norms, embeddings, MTP head | BF16 (unquantized) | 206 |
tatsu-lab/alpaca
enable_thinking=False, sequence length 2048 β used only
for the NVFP4 activation global scales and the static FP8 KV-cache scales, keeping calibration on
the activation distribution the abliterated model actually produces.llmcompressor (QuantizationModifier, two config groups + kv_cache_scheme),
format: mixed-precision. Split: 168 NVFP4 / 233 FP8 / 206 BF16 linears.vLLM serves this through the compressed-tensors path: the FP4 layers use FP4 tensor cores on
Blackwell, while the FP8 layers run on Hopper-class and newer.
Abliteration is a weight edit shared across all releases of this model, so the refusal
behavior of this checkpoint tracks the BF16 / FP8 builds. On the byte-identical-scheme
FP8 build, harmful-prompt
refusal collapses from 64β99% (base) to 0β6% (thinking off) and β€ 1.7% (thinking on),
while benign over-refusal drops (XSTest-safe 5.6% β 0.4%) and capability stays within Β±1.3 pts
of the base (MMLU 84.3 β 84.7, MMLU-Pro 77.6 β 76.8, GSM8K 90.0 β 88.7, CMMLU 81.4 β 80.8). See
that model card for the full tables.
Quant-specific numbers pending. Capability-retention and perplexity for this NVFP4 + FP8 mixed-precision checkpoint have not yet been measured β the FP4 layers require Blackwell FP4 tensor cores to run natively, and this build is released for evaluation on that hardware. Numbers will be added here once benchmarked. As a mixed 4-bit/8-bit checkpoint it is expected to trade a little accuracy for size versus the FP8 build; the dynamic split (only the less-sensitive FFN layers at FP4) is designed to keep that loss small.
The vision tower is preserved β all 167 visual.* weight tensors are kept in BF16 and the
merger / image + video preprocessor configs are intact, so this stays a full vision-language model
(Qwen3_5ForConditionalGeneration), a drop-in for the base. Abliteration only edits the
language-model residual writers, so image understanding is architecturally unaffected (and
image-conditioned refusals are reduced along with text ones). Serve without
--language-model-only to use vision.
Requires a recent vLLM (β₯ 0.27, with compressed-tensors). The FP4 layers need a Blackwell
GPU (B200 / GB200 / RTX 50-series) for native FP4 tensor cores; the FP8 layers run on Hopper-class
and newer.
docker run -d --name qwen38-uncensored-nvfp4 --gpus all --ipc=host --shm-size=8g \
-v /path/to/Qwen3.8-27B-Uncensored-NVFP4:/model:ro \
-p 8000:8000 vllm/vllm-openai:v0.27.1 \
--model /model --served-model-name Qwen3.8-27B-Uncensored \
--speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
--gpu-memory-utilization 0.9 \
--max-model-len 262144 --max-num-seqs 96 \
--trust-remote-code \
--reasoning-parser qwen3 \
--enable-auto-tool-choice --tool-call-parser qwen3_coder
The mixed-precision
quantization_config(including the FP8kv_cache_scheme) is read fromconfig.jsonβ do not pass--quantizationor--kv-cache-dtype.--speculative-config mtpenables the preserved MTP draft head.
Thinking is on by default (Qwen3.8). Toggle it per request via chat_template_kwargs; the
reasoning trace is returned in the reasoning field (--reasoning-parser qwen3).
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
model="Qwen3.8-27B-Uncensored",
messages=[{"role": "user", "content": "Prove that sqrt(2) is irrational."}],
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
print(resp.choices[0].message.reasoning) # thinking trace
print(resp.choices[0].message.content) # final answer
Standard OpenAI tools + assistant tool_calls + role: tool result messages are supported,
including multi-turn (feed the tool result back for a follow-up answer). Parsed by
--tool-call-parser qwen3_coder.
Served on OrcaRouter through the OpenAI-compatible
gateway (262K context, tools + reasoning). Grab an API key at
orcarouter.ai (sk-orca-...).
from openai import OpenAI
client = OpenAI(base_url="https://api.orcarouter.ai/v1", api_key="sk-orca-...")
resp = client.chat.completions.create(
model="qwen/qwen3.8-27b",
messages=[{"role": "user", "content": "Hello!"}],
)
print(resp.choices[0].message.content)
Software
compressed-tensors (Qwen3.5/3.8 support) β e.g. vllm/vllm-openai:v0.27.1.Compute
Memory
Throughput / concurrency
--max-num-seqs and the KV cache that fits after
weights are loaded. The MTP draft head gives a large decode speedup on real workloads.Qwen3.8-27B.Apache 2.0, inherited from the base model
Qwen/Qwen3.8-27B. Abliteration and quantization do
not change the underlying license obligations.
lm_head.weight_scale)Earlier revisions failed to load in vLLM with:
ValueError: There is no module or parameter named 'lm_head.weight_scale' in Qwen3_5ForCausalLM.
The available parameters belonging to lm_head (ParallelLMHead) are: {'lm_head.weight'}
Cause: the output head (lm_head) had been quantized to FP8, so the checkpoint shipped a
lm_head.weight_scale tensor. vLLM's Qwen3_5ForCausalLM always builds lm_head as an
unquantized ParallelLMHead (only a weight parameter), leaving the extra scale with no
destination and aborting the load.
Fix: lm_head is now kept unquantized β restored to the original BF16 weight, lm_head.weight_scale
removed, and lm_head moved to the quantization ignore list (matching the INT8 build). Only
config.json, model.safetensors.index.json, and model-00005-of-00005.safetensors changed; all
other tensors (FP4 body, FP8 layers, FP8 KV scales, MTP head) are byte-for-byte identical.
A single-file NInfer V3 (.ninfer) artifact of
orcarouter/Qwen3.8-27B-Uncensored-NVFP4,
built for an NVIDIA RTX 5090 (sm_120a), for Infernix
(recommended) and NInfer. No conversion step at load time:
point infernix-serve (or ninfer-serve) at the .ninfer file and run.
This repository is the NInfer form of the orcarouter checkpoint above, an abliterated (refusal-removed) mixed NVFP4/FP8 build of Qwen3.8-27B. Its NVFP4 and FP8 weights are carried over bit-exactly for every transformer linear, and two speculative-decoding components (DFlash2 draft model, indexed proposal head) are added so the artifact runs with the engines' fastest decode paths.
Recommended engine: Infernix, my engine grown from NInfer, which replaces my earlier ninfer-custom fork. In an agentic coding replay on the 27B it served 90 % of prompt tokens from its prefix cache and reached the first token in about half NInfer's average time, and it prefills long prompts 1.3-1.5Γ faster, at the same quality (benchmarks, measured on NInfer's official 27B artifact). It also runs Qwen3.8-Flash-Next. The artifact still works with NInfer itself and with ninfer-custom.
Disclaimer (inherited from the source model). The orcarouter checkpoint has had its safety alignment substantially removed by abliteration, so it will comply with requests the original Qwen3.8-27B would refuse. The source card releases it strictly for legitimate research (interpretability, AI-safety and refusal-mechanism study, red-teaming, robustness evaluation) and says you assume full responsibility for how you use it and what it generates; add your own safety and moderation layers before exposing it to other users. The abliteration was done by orcarouter, not by me; this repository only changes the storage format. The full source card, including its disclaimer, is reproduced at the bottom.
| Component | Contents |
|---|---|
text | Qwen3.5-family hybrid backbone: 64 layers (Gated Delta-Net linear attention + full attention every 4th layer), hidden 5120, vocab 248,320, native context 262,144 |
vision | qwen3_5_vision tower (depth 27) for image/video input |
mtp | Qwen3.5 MTP companion (the abliterated head from the source) for MTP speculative decoding |
dflash2 | 5-layer sliding-attention draft model (window 2048, selector rank 256 / top-16) β the --spec dflash2 path |
proposal | Indexed 131,072-row proposal head gathered from the source output head β enables cheap copy/proposal drafting via --lm-head-draft |
Format census of the 1,072 prepared weight objects: 112 NVFP4 (MLP projections of layers 0β55, imported bit-exact), 146 FP8 (e4m3fn, row-scale; 144 attention, GDN and last-eight-layer MLP objects imported bit-exact, plus the re-quantised embedding and output head), 579 BF16, 96 FP32, 55 Q4, 54 Q5, 28 Q8 and 1 Q6 (grouped, FP16 scales), and 1 INT32 proposal index table.
The artifact was built with the NInfer V3 converter using the qwen3_8_27b_nvfp4_orcarouter recipe
(tools/convert/official_recipes.py)
on CPU (~12 min), with components text,vision,mtp,dflash2 and --proposal --proposal-rows 131072.
qwen3_8_27b_nvfp4-uncensored.ninfer.conversion.json, included here, is the full per-tensor
conversion report. Sources and treatment:
llmcompressor
compressed-tensors mixed-precision checkpoint, per its model card) β the dominant source.
import_encoded): the packed FP4 weights and their block scales are carried
over unchanged, with no re-quantisation round-trip. At runtime these drive NInfer's W4A4 NVFP4
kernels, which dequantise weights on the fly against the stored scales and quantise
activations in-kernel.fp8_row_maxabs), the only weights of the text model that are re-quantised. The conversion
check measured the maximum relative dequantisation error against the BF16 source at 0.034 for
the embedding and 0.023 for the output head (bound: 0.07, the FP8 E4M3 half-ULP).dt_bias/a_log and the other non-projection tensors keep their source BF16/FP32 formats.--spec dflash2 path runs
on this backbone. It was not retrained for the abliterated weights.--lm-head-draft serves copy-heavy proposals from.generation_config.json and the
preprocessor configs carry the same values as the source).Verification. The converter's own checks passed for every quantisation family, and an exhaustive replay of the deterministic conversion pipeline from the source files (2026-09-27) byte-compared all 1,072 weight objects plus the auxiliary and resource objects against the artifact: zero differences. Beyond that I did not measure quality (no perplexity or benchmark numbers for this artifact), and the source card also lists the quantised checkpoint's capability retention as not yet measured.
Requires an NVIDIA RTX 50-series GPU (built and tuned on an RTX 5090; the weights take ~21.4 GiB of VRAM). Prebuilt Windows executables of Infernix are on its releases page and need an NVIDIA driver of 580 or later; on Linux, build from source (quick start).
set INFERNIX_CUDA_SYNC=blocking
infernix-serve.exe qwen3_8_27b_nvfp4-uncensored.ninfer --host 127.0.0.1 --port 8080 --max-context 210000 --max-concurrency 2 --spec dflash2 --draft-tokens 7 --lm-head-draft --ngram-draft-tokens 15 --ngram-min-match 12 --kv-dtype int8 --preserve-thinking --host-context-mib 52000 --pending-timeout-ms 900000 --prefill-chunk 4096 --kv-capacity auto --vram-headroom-mib 0 --log-colours on --ngram-archive-mib 4096 --ngram-session-mib 256 --ngram-native-sessions --request-log-jsonl log.json --default-thinking-budget 16384 --thinking-budget-message "Considering the limited time available to the user, I must stop thinking now. Time to act:" --tolerant-tool-calls --prefix-cache-file prefixes-uncensored.cache --vision --vision-offload on
On a single 32 GB GPU stop any other resident model first.
The command runs DFlash2 speculative decoding with 7 draft tokens, the indexed proposal head
(--lm-head-draft) and n-gram copy drafting, int8 KV, the hybrid prefix cache with a
52,000 MiB tier in system RAM (--host-context-mib) that is saved to and restored from
--prefix-cache-file across restarts, a thinking budget of 16,384 tokens, and recovery of
slightly malformed tool calls (--tolerant-tool-calls). --vision enables image and video input,
and --vision-offload on keeps the vision tower in system RAM instead of VRAM, which frees about
0.8 GiB of VRAM for the KV cache. INFERNIX_CUDA_SYNC=blocking stops the server keeping one CPU
core at 100% while it runs, at a cost of about 1% in output tokens per second (measured on
ninfer-custom). infernix-serve.exe --help lists every option.
With NInfer: the artifact also runs on NInfer and on
ninfer-custom with ninfer-serve and the same
command; drop any option that engine's ninfer-serve --help does not list, and use
NINFER_CUDA_SYNC in place of INFERNIX_CUDA_SYNC.
Tested on 2026-10-09 on an RTX 5090 (Windows) with the Infernix windows-2026.10.09 release:
the command above (on another port) started with this artifact and served a chat request, a
streamed request, a tool call, Anthropic Messages and Responses requests, and an image request
(which answered "red and blue" for a red image with a blue stripe). An earlier test with
ninfer-custom (2026-10-01) decoded at about 265β295 tokens per second in a handful of requests, with
DFlash2 accepting about 45β53% of drafted tokens. These are smoke tests, not benchmarks; I have not
measured speed or quality against the other artifacts.
The following is the README of the source checkpoint
orcarouter/Qwen3.8-27B-Uncensored-NVFP4, copied from its source repository for reference.
license: apache-2.0 base_model: Qwen/Qwen3.8-27B base_model_relation: quantized pipeline_tag: image-text-to-text library_name: transformers language:
An abliterated (refusal-removed) & dynamic mixed-precision NVFP4 + FP8 build of Qwen's Qwen3.8-27B β for Blackwell FP4
One Gateway. Every Model. β Route Smarter Β· Ship Safer Β· Spend Less.
Run via API Β· API Endpoint Β· Website Β· Model Catalog Β· Model Card Β· GitHub Β· Discord Β· X
An abliterated (refusal-removed) and dynamic mixed-precision NVFP4 + FP8 quantized build of
Qwen/Qwen3.8-27Bβ a 27B-parameter dense, hybrid-attention (Gated DeltaNet linear + full attention) native vision-language model with flexible thinking control, tool-calling, and an MTP speculative-decoding head. This build removes the safety refusal direction, then quantizes the bulk feed-forward layers to 4-bit NVFP4 while keeping precision-sensitive layers and the KV cache at FP8, so accuracy is better preserved than uniform W4A4. 262K context, tools + reasoning + MTP + vision preserved. Browse all models in the OrcaRouter Model Catalog.Sibling releases: Β β’Β
Qwen3.8-27B-Uncensoredβ BF16 source Β β’ΒQwen3.8-27B-Uncensored-FP8β block-FP8 for vLLM Β β’ΒQwen3.8-27B-Uncensored-GGUFβ 2-bitβ16-bit for llama.cpp Β β’ΒQwen3.8-27B-Uncensored-MLXβ MLX for Apple Silicon (2 / 4 / 8-bit).
This model has had its safety alignment substantially removed via abliteration (orthogonalizing the refusal direction out of the residual stream). As a direct consequence:
Qwen3.8-27B would refuse. It has no meaningful built-in guardrails.By downloading or using this model you acknowledge and accept the above.
| Base model | Qwen/Qwen3.8-27B |
| Architecture | Qwen3_5ForConditionalGeneration β 64 layers, hidden 5120, hybrid Gated DeltaNet (48 linear-attention + 16 full-attention, interval 4), native VL tower + MTP head |
| Modification | Abliteration (refusal-direction removal) then dynamic mixed-precision NVFP4 + FP8 quantization |
| Quantization | Mixed-precision (compressed-tensors) β NVFP4 (W4A4) on the FFN of layers 0β55, FP8 (W8A8 dynamic) on attention / GDN projections / the last 8 layers' FFN / lm_head, and a static FP8 KV cache |
| Format | safetensors, resharded to β€ 5 GB shards (5 + 1 shards, 23.4 GB, 1968 tensors) |
| Precision | FP4 E2M1 group-16 (+ FP8-E4M3 block scale + FP32 global scale) and FP8 E4M3; vision tower / norms / GDN in_proj_a/b / embeddings / MTP head kept in BF16 |
| Preserved | Full vision-language tower and MTP speculative-decoding head (drop-in for the base) |
| Context | 262,144 tokens |
Refusal-direction removal following Arditi et al. (2024), Refusal in Language Models Is
Mediated by a Single Direction. A single refusal direction r (k = 1) is estimated as the
massive-activationβmasked mean-difference of harmful β harmless last-token residuals at
layer 38 (round(0.6 Γ 64)), on AdvBench (harmful) vs Alpaca (harmless). r is then
orthogonalized out of every residual-writing matrix β W' = W β r(rα΅W) β computed in
float32:
| Component | matrices edited |
|---|---|
self_attn.o_proj (16 full-attention layers + MTP) | 17 |
linear_attn.out_proj (48 linear-attention / GDN layers) | 48 |
mlp.down_proj (64 layers + MTP) | 65 |
embed_tokens (row space) | 1 |
| Total | 131 |
The vision tower is untouched and the MTP head is abliterated consistently with the
main model, so speculative decoding keeps working. This is the same abliterated BF16 base as the
FP8,
GGUF and
BF16 releases β only the
quantization differs.
Rather than quantizing every linear uniformly, precision-sensitive layers are kept at FP8 while only the bulk feed-forward layers go to 4-bit:
| Component | Precision | # linears |
|---|---|---|
MLP gate/up/down, layers 0β55 | NVFP4 β FP4 E2M1, group 16, FP8-E4M3 block scale + FP32 global scale (W4A4) | 168 |
self_attn.{q,k,v,o}_proj, GDN in_proj_qkv/in_proj_z/out_proj | FP8 β W8A8, per-channel weight, dynamic per-token activation | 208 |
MLP gate/up/down of the last 8 layers (56β63), lm_head | FP8 (W8A8 dynamic) | 25 |
| KV cache | FP8 β static, per-tensor | β |
Vision tower, GDN in_proj_a/in_proj_b, all norms, embeddings, MTP head | BF16 (unquantized) | 206 |
tatsu-lab/alpaca
enable_thinking=False, sequence length 2048 β used only
for the NVFP4 activation global scales and the static FP8 KV-cache scales, keeping calibration on
the activation distribution the abliterated model actually produces.llmcompressor (QuantizationModifier, two config groups + kv_cache_scheme),
format: mixed-precision. Split: 168 NVFP4 / 233 FP8 / 206 BF16 linears.vLLM serves this through the compressed-tensors path: the FP4 layers use FP4 tensor cores on
Blackwell, while the FP8 layers run on Hopper-class and newer.
Abliteration is a weight edit shared across all releases of this model, so the refusal
behavior of this checkpoint tracks the BF16 / FP8 builds. On the byte-identical-scheme
FP8 build, harmful-prompt
refusal collapses from 64β99% (base) to 0β6% (thinking off) and β€ 1.7% (thinking on),
while benign over-refusal drops (XSTest-safe 5.6% β 0.4%) and capability stays within Β±1.3 pts
of the base (MMLU 84.3 β 84.7, MMLU-Pro 77.6 β 76.8, GSM8K 90.0 β 88.7, CMMLU 81.4 β 80.8). See
that model card for the full tables.
Quant-specific numbers pending. Capability-retention and perplexity for this NVFP4 + FP8 mixed-precision checkpoint have not yet been measured β the FP4 layers require Blackwell FP4 tensor cores to run natively, and this build is released for evaluation on that hardware. Numbers will be added here once benchmarked. As a mixed 4-bit/8-bit checkpoint it is expected to trade a little accuracy for size versus the FP8 build; the dynamic split (only the less-sensitive FFN layers at FP4) is designed to keep that loss small.
The vision tower is preserved β all 167 visual.* weight tensors are kept in BF16 and the
merger / image + video preprocessor configs are intact, so this stays a full vision-language model
(Qwen3_5ForConditionalGeneration), a drop-in for the base. Abliteration only edits the
language-model residual writers, so image understanding is architecturally unaffected (and
image-conditioned refusals are reduced along with text ones). Serve without
--language-model-only to use vision.
Requires a recent vLLM (β₯ 0.27, with compressed-tensors). The FP4 layers need a Blackwell
GPU (B200 / GB200 / RTX 50-series) for native FP4 tensor cores; the FP8 layers run on Hopper-class
and newer.
docker run -d --name qwen38-uncensored-nvfp4 --gpus all --ipc=host --shm-size=8g \
-v /path/to/Qwen3.8-27B-Uncensored-NVFP4:/model:ro \
-p 8000:8000 vllm/vllm-openai:v0.27.1 \
--model /model --served-model-name Qwen3.8-27B-Uncensored \
--speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
--gpu-memory-utilization 0.9 \
--max-model-len 262144 --max-num-seqs 96 \
--trust-remote-code \
--reasoning-parser qwen3 \
--enable-auto-tool-choice --tool-call-parser qwen3_coder
The mixed-precision
quantization_config(including the FP8kv_cache_scheme) is read fromconfig.jsonβ do not pass--quantizationor--kv-cache-dtype.--speculative-config mtpenables the preserved MTP draft head.
Thinking is on by default (Qwen3.8). Toggle it per request via chat_template_kwargs; the
reasoning trace is returned in the reasoning field (--reasoning-parser qwen3).
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
model="Qwen3.8-27B-Uncensored",
messages=[{"role": "user", "content": "Prove that sqrt(2) is irrational."}],
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
print(resp.choices[0].message.reasoning) # thinking trace
print(resp.choices[0].message.content) # final answer
Standard OpenAI tools + assistant tool_calls + role: tool result messages are supported,
including multi-turn (feed the tool result back for a follow-up answer). Parsed by
--tool-call-parser qwen3_coder.
Served on OrcaRouter through the OpenAI-compatible
gateway (262K context, tools + reasoning). Grab an API key at
orcarouter.ai (sk-orca-...).
from openai import OpenAI
client = OpenAI(base_url="https://api.orcarouter.ai/v1", api_key="sk-orca-...")
resp = client.chat.completions.create(
model="qwen/qwen3.8-27b",
messages=[{"role": "user", "content": "Hello!"}],
)
print(resp.choices[0].message.content)
Software
compressed-tensors (Qwen3.5/3.8 support) β e.g. vllm/vllm-openai:v0.27.1.Compute
Memory
Throughput / concurrency
--max-num-seqs and the KV cache that fits after
weights are loaded. The MTP draft head gives a large decode speedup on real workloads.Qwen3.8-27B.Apache 2.0, inherited from the base model
Qwen/Qwen3.8-27B. Abliteration and quantization do
not change the underlying license obligations.
lm_head.weight_scale)Earlier revisions failed to load in vLLM with:
ValueError: There is no module or parameter named 'lm_head.weight_scale' in Qwen3_5ForCausalLM.
The available parameters belonging to lm_head (ParallelLMHead) are: {'lm_head.weight'}
Cause: the output head (lm_head) had been quantized to FP8, so the checkpoint shipped a
lm_head.weight_scale tensor. vLLM's Qwen3_5ForCausalLM always builds lm_head as an
unquantized ParallelLMHead (only a weight parameter), leaving the extra scale with no
destination and aborting the load.
Fix: lm_head is now kept unquantized β restored to the original BF16 weight, lm_head.weight_scale
removed, and lm_head moved to the quantization ignore list (matching the INT8 build). Only
config.json, model.safetensors.index.json, and model-00005-of-00005.safetensors changed; all
other tensors (FP4 body, FP8 layers, FP8 KV scales, MTP head) are byte-for-byte identical.