Qwen3.6-27B-tq3-g32

Full 3-bit TurboQuant quantization of Qwen/Qwen3.6-27B (group size 32), produced with TurboQuant-MLX.

A local coding model that actually fits a 48 GB Mac. Qwen3.6-27B is a dense, long-context model positioned by Qwen as a strong coder (competitive on SWE-bench Verified / SWE-bench Pro). At 3-bit it shrinks from ~55 GB BF16 to ~13 GB on disk / ~17.5 GB peak at runtime — leaving real headroom on a 48 GB machine for long context, so you can point an IDE (Cursor, VS Code via an OpenAI-compatible endpoint) at it for local "vibe coding" / agentic coding without a 64 GB+ box or the cloud.

Model Details

• Base model: Qwen/Qwen3.6-27B — dense (not MoE), model_type: qwen3_5
• Architecture: 64 layers, hidden size 5120, 24 attention heads / 4 KV heads, head_dim 256, intermediate 17408, vocab 248320, 262 144-token context. Hybrid attention: 48 GatedDeltaNet linear-attention layers + 16 full-attention layers (a 3:1 ratio, full_attention_interval=4) — most layers use constant-memory linear attention, so long context stays cheap.
• Quantization: TurboQuant (randomized Hadamard rotation + Lloyd-Max codebook), uniform 3-bit, group size 32
  • All 497 Linear projections (attention q/k/v/o, GatedDeltaNet projections, MLP gate/up/down, lm_head) → 3-bit
  • This is a dense model, so there are no experts to stream — it runs fully resident.
• Size: ~13 GB across 3 shards (vs ~55 GB BF16)

Note: this is an instruct / chat model with a thinking mode — its chat template emits a reasoning trace before the answer. The streaming generator applies the chat template automatically.

Will it fit my Mac?

Don't guess, and don't download 14.4 GB to find out — ask first (needs turboquant-mlx-full >= 0.15.1):

turboquant-plan --model manjunathshiva/Qwen3.6-27B-tq3-g32

It reads only this repo's safetensors headers over the network (a couple of hundred KB, a few seconds — the weights are never fetched), then projects the peak against your machine and prints the flags to use:

Model
  weights (exact)      14.4 GB
  KV cache             64.0 KB/token  (hybrid: 16/64 full-attention)

Verdict: <resident | needs a wired-limit raise | streaming>
Recommended: <the exact flags for your Mac>

Planning for a machine you're not sitting at — deciding whether a 16 GB mini or a 64 GB Mac can take this — is what --wired-gb / --ram-gb are for:

turboquant-plan --model manjunathshiva/Qwen3.6-27B-tq3-g32 \
    --wired-gb 10.5 --ram-gb 16 --context 8192

turboquant-doctor runs the same projection plus a readiness check; both take --json. The projection is calibrated against real measurements on a 16 GB Mac mini rather than estimated from theory — on the 9.4 GB ternary 35B it predicts a 10.44 GB peak where that machine measures 10.42.

Why this build

The 27B is the sweet spot for local coding: big enough to be genuinely good at code, small enough that 3-bit TurboQuant drops it under the ~18 GB runtime mark. Because it's dense, it doesn't need expert streaming (unlike the 100B+ MoE TurboQuant models) — it loads once and stays in RAM, so an HDD/SSD only matters for load time. The hybrid GatedDeltaNet attention (only 16 of 64 layers keep a growing KV cache) keeps long-context memory growth modest, which is what makes large coding contexts practical on 48 GB.

Quality

Coding sanity check (3-bit, --temp 0.7, on a 64 GB Mac): asked for merge_intervals, it derived the correct sort → sweep → max-merge algorithm, handled the empty case, and self-verified its own example ([[1,3],[2,6],[8,10],[15,18]] → [[1,6],[8,10],[15,18]]). Coherent and correct.

Requirements

# macOS with Apple Silicon (M1/M2/M3/M4)
pip install turboquant-mlx-full mlx-lm

Quick Start

python -m turboquant_mlx.generate \
    --model ~/path/to/Qwen3.6-27B-tq3-g32 \
    --prompt "Write a Python function that merges overlapping intervals." \
    --max-tokens 512 \
    --temp 0.7

Serve over an OpenAI-compatible API (Cursor / VS Code)

⚠️ Stock mlx_lm's load() / mlx_lm.server cannot load a TurboQuant model — the quantization.mode = "turboquant" field raises KeyError: 'turboquant'. Use turboquant-serve, which wraps mlx_lm.server and patches the loader so the weights load through the PolarQuant path, then exposes the standard OpenAI endpoints. This is how you wire the model into an IDE for local coding.

# Serve directly from the Hugging Face Hub
turboquant-serve --model manjunathshiva/Qwen3.6-27B-tq3-g32 --port 8080

# ...or from a local copy
turboquant-serve --model ./Qwen3.6-27B-tq3-g32 --port 8080

Call it like any OpenAI-compatible endpoint (the model field must match the --model string). Use a generous max_tokens — this is a thinking model, so the reasoning trace and the final answer share the budget (mlx-lm splits them into message.reasoning and message.content):

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "manjunathshiva/Qwen3.6-27B-tq3-g32",
    "messages": [{"role": "user", "content": "Write a Python function that merges overlapping intervals."}],
    "max_tokens": 2048,
    "temperature": 0.7
  }'

From Python via the OpenAI SDK:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
resp = client.chat.completions.create(
    model="manjunathshiva/Qwen3.6-27B-tq3-g32",
    messages=[{"role": "user", "content": "Write a Python function that merges overlapping intervals."}],
    max_tokens=2048,
    temperature=0.7,
    stream=True,
)
for chunk in resp:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

In Cursor / VS Code, point the IDE's custom "OpenAI-compatible" base URL at http://localhost:8080/v1 and set the model to manjunathshiva/Qwen3.6-27B-tq3-g32.

All mlx_lm.server flags forward unchanged (turboquant-serve --help) — including --draft-model / --num-draft-tokens for speculative decoding. Note: mlx_lm.server is for development / local use — no authentication or rate limiting.

Long context — KV cache compression

Only the 16 full-attention layers keep a growing KV cache (the 48 linear-attention layers use a fixed-size state), so this model is already KV-light for long context. For very long prompts you can compress the KV cache further (mixed precision recommended: K at 8-bit, V at 3-bit):

python -m turboquant_mlx.generate \
    --model ~/path/to/Qwen3.6-27B-tq3-g32 \
    --prompt "$(cat your_long_codebase_context.txt)" \
    --max-tokens 512 --temp 0.7 \
    --kv-k-bits 8 --kv-v-bits 3

How It Works

TurboQuant applies:

1. Hadamard rotation — randomized ±1 scaling to decorrelate / Gaussianize weights before quantization
2. Lloyd-Max codebook — optimal scalar quantization for the Gaussian distribution (via k-means)
3. Group-wise scaling — per-group float16 scales for precision

This achieves better quality than standard affine quantization at the same bit-width.

Reproducing the conversion

python -m turboquant_mlx.convert \
    --hf-path Qwen/Qwen3.6-27B \
    --mlx-path /path/to/Qwen3.6-27B-tq3-g32 \
    --bits 3 --group-size 32 --streaming

--streaming writes each quantized layer to a shard and frees it, so the full model converts in a few GB of RAM. (TurboQuant-MLX ≥ 0.6.2 also forces the disk read ahead of GPU compute, so conversion works even off slow USB storage without tripping the Metal GPU watchdog.)

License

Apache 2.0 (same as the base model). Quantization tooling: TurboQuant-MLX.

Copyright 2026 Manjunath Janardhan.

Citation

@article{zandieh2025turboquant,
  title={TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate},
  author={Zandieh, Amir and Daliri, Majid and Hadian, Majid and Mirrokni, Vahab},
  year={2025},
  eprint={2504.19874},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2504.19874}
}