vllm-ampere-optimized

An Ampere fork of vLLM v0.23.0 (+ FlashInfer v0.6.12) that un-gates W4A8 (int4 weights + int8 activations) and adds native int8 kernels upstream restricts to Hopper. Built from source for sm_80 (A100) and sm_86 (RTX 3090 / A40 / A6000 / A10).

Image: ghcr.io/avesed/vllm-ampere-optimized

Why

W4A8 is a strong serving quant — int4 weights cut decode bandwidth, int8 activations speed up prefill. Marlin can run it on Ampere, but vLLM gates its W4A8 path to Hopper: on an Ampere GPU a W4A8 checkpoint crashes at load, so Ampere users are stuck on W4A16. This fork routes W4A8 through Marlin so it runs.

What's in it

• W4A8 on Ampere (patches/0001, upstream #38066) — int4-weight + int8-act through Marlin.
• int8 8-row Marlin decode tile (patches/0002) — completes the W4A8 small-batch decode path.
• AOT-compile cache-key fix (patches/0003) — keys the torch.compile cache on the quant scheme.
• int8-act opt-in flag + MoE support (patches/0005–0006) — --marlin-input-dtype int8 (or env VLLM_MARLIN_INPUT_DTYPE=int8) turns a W4A16 checkpoint into W4A8 at serve time, for dense and MoE.
• Vendored famp Marlin kernel (flashampere/marlin/) — the fork owns its W4A8/W4A16 Marlin GEMM as a plugin-selected .so (bit-exact vs stock, built in the image for sm_80+sm_86), so the int8 path survives upstream refactors.
• flashampere attention backend (flashampere/, opt-in VLLM_FLASHAMPERE=1) — composable Ampere attention legs: fp16-accumulate PV prefill (GeForce RTX-30 only; +1–4% long-context TTFT), a vendored XQA decode kernel for head_dim-512 (gemma-4) and MTP spec-verify, with bit-faithful fallback to stock FA. A per-model-architecture registry auto-applies validated defaults.
• DSpark / DFlash speculative decoding — native --speculative-config '{"method":"dspark",...}' serving of block-diffusion draft heads (DeepSeek DSpark / z-lab DFlash); ready-made head: Avesed/Qwen3.6-27B-DSpark.

vllm/ and flashinfer/ carry the edits baked in; patches/ + scripts/revendor.sh replay them on an upstream bump, and scripts/build_image_source.sh builds + pushes the image.

Results

Stock vLLM won't load W4A8 on any Ampere GPU — the fork is the only way to run it. Numbers below are W4A16 → W4A8 on the same fork engine, tok/s (int4 g32 AWQ+mse, cudagraph):

• The int8 prefill win is a consumer-sm_86 effect — those cards' fp16 tensor (FP32 accumulate) is half-rate, so int8 is a ~4× compute lever. A100 (sm_80) fp16 is full-rate → int8 prefill ~0 (dense) / +8% (MoE). The W4A8 enabler + int4-weight decode/VRAM savings hold on every Ampere card.
• No-NVLink multi-GPU → -pp 2 -tp 1 — TP's all-reduce eats ~half of prefill (it shrinks the 35B int8 gain to +5%).
• Quality: decode is W4A16-parity; int8 activations cost ~zero accuracy — GSM8K (thinking) 9B W4A16 81.6% / W4A8 85.6% (N=250); 35B-A3B W4A8 GSM8K 95.8%, MMLU-Pro 80.5%. The fork's W4A16 is byte-identical to stock.
• DiffusionGemma (block-diffusion dLLM, the row above): prefill (8k) + single-stream gen, canvas 256. Unlike AR decode (≈flat under int8), the dLLM's generation is compute-bound too (every denoise step is prefill-like), so it gains as well (178 → 213, +20%). int8-act also beats NVFP4 (176) — no native FP4 kernel below sm_89, so it dequant-emulates to bf16 (memory-only, ≈ W4A16 speed) — at matched accuracy: W4A8 GSM8K 95.7% / MMLU-Pro 76.4%; NVFP4 96.0% / 77.8% (N=1000/500). Two serve requirements: --dtype bfloat16 (under int8, Gemma's large activations overflow fp16 — down-proj GEMM dequant > 65504 → inf → NaN → blank output) and --attention-backend TRITON_ATTN (the diffusion mixed causal/bidirectional mask isn't supported by flash-attn (wants FA4) or FlashInfer — the int8 win is the MoE GEMM alone). Run the cyankiwi W4A16 ckpt with VLLM_MARLIN_INPUT_DTYPE=int8 --dtype bfloat16 --attention-backend TRITON_ATTN over the chat endpoint.

Use

docker run --gpus all -p 8000:8000 \
  ghcr.io/avesed/vllm-ampere-optimized:latest \
  --model Avesed/Qwen3.6-27B-INT4-W4A16 --marlin-input-dtype int8 --pipeline-parallel-size 2 --max-model-len 8192

(cu130 image needs NVIDIA driver ≥ 580.65. With NVLink use --tensor-parallel-size 2; single GPU, drop both. On a no-NVLink multi-GPU box, prefer -pp 2, or add --disable-custom-all-reduce if you use -tp 2 — custom all-reduce over PCIe + expandable_segments can crash at startup.)

Run a plain W4A16 checkpoint as W4A8 by adding --marlin-input-dtype int8 (dense or MoE).

No Docker? A from-source wheel (sm_80+sm_86) is on Releases — pip install it (needs torch 2.11

• CUDA 13). Enable W4A8 with --marlin-input-dtype int8.

• Ready-made quants — huggingface.co/Avesed:
  • Qwen3.6-27B: INT4-W4A16 · INT8-W8A8 — int4: GSM8K 96.8% / MMLU-Pro 82.4%
  • Qwen3.6-35B-A3B (MoE): INT4-W4A16 · INT8-W8A8 — int4: GSM8K 96.8% / MMLU-Pro 80.2%
• Quantize your own: python quantize/quantize_w4a8.py <hf-model> <out-dir> — best quality = the AWQ + mse + g32 recipe in quantize/README.md

Credits

Built on vllm-project/vllm (Apache-2.0); W4A8 enablement follows upstream #38066.