Asymmetric-expert TurboQuant quantization of Qwen/Qwen3.6-35B-A3B: 3-bit attention, ternary (1.58-bit) expert up/gate_proj + 4-bit expert down_proj (group size 64) — produced with TurboQuant-MLX.
This is the agent-capable sibling of the 9.4 GB pure-ternary build. Spending 4 bits on just the expert down_proj (one of the three expert matrices) costs +3.2 GB (12.6 GB total) and is the difference between failing and passing multi-step tool-use: in a fix-the-failing-test task under Opencode this build went 3/3, where pure ternary went 0/4 and a 3-bit down_proj variant still went 0/3.
qwen3_5_moe)lm_head → 3-bitgate_proj / up_proj → ternary {-c, 0, +c}, base-3 trit-packed (~1.6 bpw)down_proj → 4-bit Lloyd-Max codebookdown_proj (and not the other two)?The expert MLP computes down_proj(silu(gate_proj(x)) * up_proj(x)). up/gate feed a gated nonlinearity that is tolerant of coarse weights, and their errors are partially averaged across the top-8 routed experts. down_proj is the write-back into the residual stream — its errors land directly on the token representation every layer. Measured on the Opencode agentic task, capability turns on between 3 and 4 bits on this one matrix: down3 fails like pure ternary, down4 completes the full observe → diagnose → edit → verify loop. (Same recipe as DwarfStar/llama.cpp lore: spend bits on down, starve up/gate.)
Requires turboquant-mlx-full ≥ 0.13 (the expert_down_bits config field is new):
pip install "turboquant-mlx-full>=0.13" mlx-lm
python -m turboquant_mlx.generate \
--model manjunathshiva/Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64 \
--prompt "Explain why the sky is blue." \
--max-tokens 512
On a 16 GB Mac mini the weights exceed the default Metal wired-memory cap (~10.5 GB), so raise it once per boot:
sudo sysctl -w iogpu.wired_limit_mb=13824 # 13.5 GiB
turboquant-plan, ≥ 0.15.1)Don't guess, and don't download 12.6 GB to find out — ask first:
turboquant-plan --model manjunathshiva/Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64
It reads only this repo's safetensors headers over the network (~240 KB, a couple of seconds — the weights are never fetched) and projects the peak against your machine, then prints the flags to use:
Projection at 21,000 tokens of context
weights 12.59 GB
KV cache 0.22 GB (10.0 KB/token, hybrid: 10/40 full-attention layers)
prefill workspace 0.09 GB (estimate, at --prefill-step-size 128)
runtime reserve 1.00 GB (buffer cache, activations, fragmentation)
----------------------------------
peak 13.89 GB of 14.40 GB usable 0.51 GB headroom
Verdict: ⚠️ RESIDENT — fits, but only after raising the Metal wired cap
Recommended:
sudo sysctl -w iogpu.wired_limit_mb=13721
--prefill-step-size 128
--kv-bits 8
Planning for a machine you're not sitting at — say, deciding whether to buy or
use a mini — is what --wired-gb / --ram-gb are for (the run above is exactly
that: a 16 GB mini, projected from a 64 GB machine). turboquant-doctor adds a
readiness check, and both take --json.
The projection is calibrated against real measurements on the mini rather than estimated from theory: it predicts a 10.44 GB peak where the 9.4 GB ternary sibling measures 10.42, and the wired limit it suggests here lands within 1% of the 13824 that this build actually runs on.
pip install "turboquant-mlx-full[serve]>=0.13"
turboquant-serve \
--model manjunathshiva/Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64 \
--kv-bits 8 --tool-syntax-greedy
--tool-syntax-greedy forces argmax inside <tool_call> blocks (keys, punctuation, tags) while leaving payload strings and the call-or-not decision sampled — cheap insurance for tool-call syntax at low bit-widths. Use temperature ~0.7; greedy-everything hurts this model's planning.
Agentic harness test (Opencode, fix-the-failing-test task, identical server/harness/prompt across builds, 64 GB M4 Max):
| Build | Experts | Size | Opencode result |
|---|---|---|---|
| pure ternary (tq3a-tqTe-g64) | 1.58-bit ×3 | 9.4 GB | ❌ 0/4 — command loops, hallucinated tool args |
| ternary + 3-bit down_proj | down3 | 11.6 GB | ❌ 0/3 |
| ternary + 4-bit down_proj (this) | down4 | 12.6 GB | ✅ 3/3 — full test → diagnose → fix → green loop |
| 3-bit everything (tq3-g32) | 3-bit | 16.4 GB | ✅ (control, 33 s) |
Validated end-to-end on a base M-series 16 GB Mac mini (model on an external SSD), serving over the OpenAI-compatible API:
| Metric | Result |
|---|---|
| Decode | 15.2–15.6 tok/s, flat from 300 to 1800 generated tokens |
| Peak memory (generate) | 13.6 GB |
| Cold prefill | 21K-token document in ~234 s (~90 tok/s, 256-token chunks) |
| Opencode agentic task | ✅ 3/3 (fix-the-failing-test: 3:37 / 2:25 / 1:54 — faster each run as the disk cache warms) |
| Follow-up-turn cache reuse | 20,480 / 21,250 tokens restored from disk checkpoints (96.4%) |
| Long-form generation | coherent with the shipped repetition_penalty: 1.05 (this repo's generation_config.json) |
The validated serving recipe (requires turboquant-mlx-full ≥ 0.13):
sudo sysctl -w iogpu.wired_limit_mb=14336 # 14 GiB; 13824 works below ~16K context
turboquant-serve \
--model manjunathshiva/Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64 \
--host 0.0.0.0 --port 8080 \
--kv-bits 8 --tool-syntax-greedy --disk-cache \
--prefill-step-size 128 \
--temp 0.7 --top-p 0.8 --top-k 20 \
--chat-template-args '{"enable_thinking": false}' \
--prompt-concurrency 1
Point Opencode (or any OpenAI-compatible agent harness) at
http://<host>:8080/v1 with model id default_model, temperature ~0.7,
and auto-discovered skills disabled. Practical context ceiling on 16 GB is
~16–18K tokens at the default wired cap; the --prefill-step-size 128 +
14 GiB wired combination above is what carries 21K+ (the serve defaults
--metal-cache-limit-gb auto and --prompt-cache-max-gb auto handle the
rest of the tight-memory story automatically).
| Build | Size | Best for |
|---|---|---|
| tq3a-tqTe-g64 | 9.4 GB | chat, drafting, Q&A on a 16 GB mini — default wired cap, no sudo |
| tq3a-tqTe-down4-g64 (this) | 12.6 GB | agent harnesses / tool loops on a 16 GB mini (needs the sysctl bump) |
| tq3-g32 | 16.4 GB | recall-critical work, exact literals; 24 GB+ machines |
python -m turboquant_mlx.convert \
--hf-path Qwen/Qwen3.6-35B-A3B \
--mlx-path Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64 \
--bits 3 --group-size 64 --ternary-experts --expert-down-bits 4
Data-free (no calibration set): randomized Hadamard rotation → per-group RMS scaling → Lloyd-Max codebook (ternary for up/gate, 16-level for down) → trit/bit packing. Per-projection bit-width is self-describing on disk via codebook length; expert_down_bits is recorded in config.json so converter and loader agree.
Asymmetric-expert TurboQuant quantization of Qwen/Qwen3.6-35B-A3B: 3-bit attention, ternary (1.58-bit) expert up/gate_proj + 4-bit expert down_proj (group size 64) — produced with TurboQuant-MLX.
This is the agent-capable sibling of the 9.4 GB pure-ternary build. Spending 4 bits on just the expert down_proj (one of the three expert matrices) costs +3.2 GB (12.6 GB total) and is the difference between failing and passing multi-step tool-use: in a fix-the-failing-test task under Opencode this build went 3/3, where pure ternary went 0/4 and a 3-bit down_proj variant still went 0/3.
qwen3_5_moe)lm_head → 3-bitgate_proj / up_proj → ternary {-c, 0, +c}, base-3 trit-packed (~1.6 bpw)down_proj → 4-bit Lloyd-Max codebookdown_proj (and not the other two)?The expert MLP computes down_proj(silu(gate_proj(x)) * up_proj(x)). up/gate feed a gated nonlinearity that is tolerant of coarse weights, and their errors are partially averaged across the top-8 routed experts. down_proj is the write-back into the residual stream — its errors land directly on the token representation every layer. Measured on the Opencode agentic task, capability turns on between 3 and 4 bits on this one matrix: down3 fails like pure ternary, down4 completes the full observe → diagnose → edit → verify loop. (Same recipe as DwarfStar/llama.cpp lore: spend bits on down, starve up/gate.)
Requires turboquant-mlx-full ≥ 0.13 (the expert_down_bits config field is new):
pip install "turboquant-mlx-full>=0.13" mlx-lm
python -m turboquant_mlx.generate \
--model manjunathshiva/Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64 \
--prompt "Explain why the sky is blue." \
--max-tokens 512
On a 16 GB Mac mini the weights exceed the default Metal wired-memory cap (~10.5 GB), so raise it once per boot:
sudo sysctl -w iogpu.wired_limit_mb=13824 # 13.5 GiB
turboquant-plan, ≥ 0.15.1)Don't guess, and don't download 12.6 GB to find out — ask first:
turboquant-plan --model manjunathshiva/Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64
It reads only this repo's safetensors headers over the network (~240 KB, a couple of seconds — the weights are never fetched) and projects the peak against your machine, then prints the flags to use:
Projection at 21,000 tokens of context
weights 12.59 GB
KV cache 0.22 GB (10.0 KB/token, hybrid: 10/40 full-attention layers)
prefill workspace 0.09 GB (estimate, at --prefill-step-size 128)
runtime reserve 1.00 GB (buffer cache, activations, fragmentation)
----------------------------------
peak 13.89 GB of 14.40 GB usable 0.51 GB headroom
Verdict: ⚠️ RESIDENT — fits, but only after raising the Metal wired cap
Recommended:
sudo sysctl -w iogpu.wired_limit_mb=13721
--prefill-step-size 128
--kv-bits 8
Planning for a machine you're not sitting at — say, deciding whether to buy or
use a mini — is what --wired-gb / --ram-gb are for (the run above is exactly
that: a 16 GB mini, projected from a 64 GB machine). turboquant-doctor adds a
readiness check, and both take --json.
The projection is calibrated against real measurements on the mini rather than estimated from theory: it predicts a 10.44 GB peak where the 9.4 GB ternary sibling measures 10.42, and the wired limit it suggests here lands within 1% of the 13824 that this build actually runs on.
pip install "turboquant-mlx-full[serve]>=0.13"
turboquant-serve \
--model manjunathshiva/Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64 \
--kv-bits 8 --tool-syntax-greedy
--tool-syntax-greedy forces argmax inside <tool_call> blocks (keys, punctuation, tags) while leaving payload strings and the call-or-not decision sampled — cheap insurance for tool-call syntax at low bit-widths. Use temperature ~0.7; greedy-everything hurts this model's planning.
Agentic harness test (Opencode, fix-the-failing-test task, identical server/harness/prompt across builds, 64 GB M4 Max):
| Build | Experts | Size | Opencode result |
|---|---|---|---|
| pure ternary (tq3a-tqTe-g64) | 1.58-bit ×3 | 9.4 GB | ❌ 0/4 — command loops, hallucinated tool args |
| ternary + 3-bit down_proj | down3 | 11.6 GB | ❌ 0/3 |
| ternary + 4-bit down_proj (this) | down4 | 12.6 GB | ✅ 3/3 — full test → diagnose → fix → green loop |
| 3-bit everything (tq3-g32) | 3-bit | 16.4 GB | ✅ (control, 33 s) |
Validated end-to-end on a base M-series 16 GB Mac mini (model on an external SSD), serving over the OpenAI-compatible API:
| Metric | Result |
|---|---|
| Decode | 15.2–15.6 tok/s, flat from 300 to 1800 generated tokens |
| Peak memory (generate) | 13.6 GB |
| Cold prefill | 21K-token document in ~234 s (~90 tok/s, 256-token chunks) |
| Opencode agentic task | ✅ 3/3 (fix-the-failing-test: 3:37 / 2:25 / 1:54 — faster each run as the disk cache warms) |
| Follow-up-turn cache reuse | 20,480 / 21,250 tokens restored from disk checkpoints (96.4%) |
| Long-form generation | coherent with the shipped repetition_penalty: 1.05 (this repo's generation_config.json) |
The validated serving recipe (requires turboquant-mlx-full ≥ 0.13):
sudo sysctl -w iogpu.wired_limit_mb=14336 # 14 GiB; 13824 works below ~16K context
turboquant-serve \
--model manjunathshiva/Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64 \
--host 0.0.0.0 --port 8080 \
--kv-bits 8 --tool-syntax-greedy --disk-cache \
--prefill-step-size 128 \
--temp 0.7 --top-p 0.8 --top-k 20 \
--chat-template-args '{"enable_thinking": false}' \
--prompt-concurrency 1
Point Opencode (or any OpenAI-compatible agent harness) at
http://<host>:8080/v1 with model id default_model, temperature ~0.7,
and auto-discovered skills disabled. Practical context ceiling on 16 GB is
~16–18K tokens at the default wired cap; the --prefill-step-size 128 +
14 GiB wired combination above is what carries 21K+ (the serve defaults
--metal-cache-limit-gb auto and --prompt-cache-max-gb auto handle the
rest of the tight-memory story automatically).
| Build | Size | Best for |
|---|---|---|
| tq3a-tqTe-g64 | 9.4 GB | chat, drafting, Q&A on a 16 GB mini — default wired cap, no sudo |
| tq3a-tqTe-down4-g64 (this) | 12.6 GB | agent harnesses / tool loops on a 16 GB mini (needs the sysctl bump) |
| tq3-g32 | 16.4 GB | recall-critical work, exact literals; 24 GB+ machines |
python -m turboquant_mlx.convert \
--hf-path Qwen/Qwen3.6-35B-A3B \
--mlx-path Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64 \
--bits 3 --group-size 64 --ternary-experts --expert-down-bits 4
Data-free (no calibration set): randomized Hadamard rotation → per-group RMS scaling → Lloyd-Max codebook (ternary for up/gate, 16-level for down) → trit/bit packing. Per-projection bit-width is self-describing on disk via codebook length; expert_down_bits is recorded in config.json so converter and loader agree.