Gemma 4 21B REAP AWQ 4-bit (thinking + vision + video)

AWQ 4-bit quantization of cerebras/Gemma-4-21B-REAP-A4B-it — Cerebras's REAP-pruned variant of Gemma 4 26B (drops 32 of 128 experts → 96, ~21B total / 3B active) — calibrated to preserve thinking, vision, and video on consumer-class GPUs. Validated on NVIDIA RTX 3090 (Ampere sm_86, 24 GB) with SGLang v0.5.11 + 16 patches.

Model details

Validation (2x RTX 3090, TP=1 / 4K, SGLang v0.5.11 + patches 023 + 028)

4/4 PASS in 12.0s via scripts/eval/validate_capabilities.py:

Two SGLang patches are required for this checkpoint to load and serve correctly:

• Patch 028 — gemma4_mm per-expert AWQ loader (maps model.layers.N.mlp.experts.E.{gate,up,down}_proj.{qweight,scales,qzeros} correctly when checkpoint stores per-expert keys).
• Patch 023 — detects whether the dense MLP is quantized from quantization_config.ignore rather than hardcoding quant_config=None for dense MLP on MoE layers.

Without both, you'll see either KeyError on per-expert keys or BF16 placeholders that never match the AWQ checkpoint.

Recommended runtime

# config.json carries architectures=Gemma4ForConditionalGeneration (multimodal route) by default.
git clone https://github.com/mattbucci/2x-3090-GA102-300-A1-sglang-inference
cd 2x-3090-GA102-300-A1-sglang-inference
./scripts/setup.sh                           # clones SGLang v0.5.11 + applies 16 patches
MODEL=mattbucci/gemma-4-21B-REAP-AWQ ./scripts/launch.sh gemma4

The gemma4 preset already bakes the working flag combo for Ampere sm_86 (head_dim=256 + FP8 KV incompat):

• --attention-backend triton (FlashInfer rejects head_dim=256)
• --kv-cache-dtype auto (FP8 E4M3 KV incompat with triton on sm_86)
• --dtype bfloat16 (SigLIP vision tower NaNs in FP16)
• --disable-cuda-graph --disable-piecewise-cuda-graph (trades cold-launch reliability for ~5-10% TPOT)
• --enable-multimodal --reasoning-parser gemma4

Why REAP-21B over the 26B base

REAP (Cerebras) prunes the 32 lowest-impact experts from the 128-expert original, dropping the weight footprint from ~5.6 GB at AWQ-4bit (26B) to ~4.4 GB (21B) while retaining most of the multimodal capability. The smaller active-param fraction (3B vs 4B) also leaves more KV pool headroom on a 24 GB 3090 at long context.

For thinking + vision the 26B base remains slightly stronger; for tighter VRAM budgets or higher-throughput inference the 21B REAP variant is the right pick. Both are content-aware on the validator's red-circle / moving-red-circle probes.

Known limitations

• Cross-stack RDNA4 cross-validation pending. R9700 (gfx1201, ROCm 7.2) port of patch 023 not yet landed; cross-validation will run after their v0.5.11 env upgrade. Until then this is Ampere-only verified.
• v2 build (-thinking-vision-v2) is a calibration disaster — 164 all-zero scale tensors in the vision tower + MoE experts (silent calibration failure when the ignore list used bare strings instead of regex). The shipped checkpoint is the v3b corrected build; do not pull older -v2 directories from sister repos.
• The 26B base supports an audio modality; this REAP-21B variant inherits whatever survived REAP pruning of the audio tower. Validated on text/image/video; audio not exercised in the 4/4 sweep.

Hardware

Validated on 2x NVIDIA RTX 3090 (GA102-300-A1, Ampere sm_86, 24 GB GDDR6X each) with CUDA 13.2 / driver 595.58 / PyTorch 2.11.0 + cu130 / SGLang v0.5.11 + 16 local patches. Fits comfortably on a single 3090 at TP=1 / 4K.

Citation

If you use this checkpoint, please cite the REAP authors and Google's Gemma 4 release:

• Cerebras REAP — github.com/CerebrasResearch/reap
• Gemma 4 — ai.google.dev/gemma/docs/gemma4