SUSHI

A local AI server for low-bit, very high quality Sushi quants, using custom EXL3-Affine hybrid converter. It is a detached fork of ddalcu's mlx-serve. Sushi runs stand-alone and also as a guest engine inside mlx-serve, Apple Silicon M1+ is required.

Model support list

Usage see recommended launch commands and coding agent usage.

• Qwen3.8-Flash-Next-Sushi-2bpw (requires 48 GB+, or 32 GB streamed)
• Qwen3.8-Flash-Next-Sushi-2.6bpw (requires 64 GB+)
• Qwen3.8-Flash-Next-Sushi-3bpw (requires 64 GB+)
• Qwen3.8-Flash-Next-Sushi-4bpw (requires 96 GB+)
• MiMo-V2.6-Flash-Sushi-2.3bpw (requires 128 GB, text and image input)
• GLM-5.3-Flash-Sushi-2bpw (requires 96 GB+, text, image and video input)
• GLM-5.3-Flash-Sushi-2.4bpw (requires 128 GB, text, image and video input)

Quality

• MiMo-V2.6-Flash-Sushi-2.3bpw scores KLD 0.0860 (top-1 agreement 91.95%)
• GLM-5.3-Flash-Sushi-2.4bpw scores KLD 0.0742 (top-1 agreement 90.33%)
• GLM-5.3-Flash-Sushi-2bpw scores KLD 0.1037 (top-1 agreement 88.18%)

Install

brew install beamivalice/tap/sushi

Update with brew upgrade sushi.

• The server listens on 127.0.0.1:12345.
• Support speculative decoding: MTP for Qwen and MiMo, DFlash2 for GLM.
• 8-bit KV cache are on by default.

Memory

GPU memory in GiB to serve one prompt that fills the whole context (8-bit KV, MTP on with --mtp-head-kv-quant; GLM with its DFlash2 assistant and vision tower). The prompt cache and the n-gram table stay on the SSD and are not counted. The Qwen columns are at the default f32 GatedDeltaNet state; --qwen-gdn 16 (bf16) takes about 0.2 GiB off each Qwen cell.

A context fits when its number is below the GPU limit you set with sudo sysctl iogpu.wired_limit_mb. Max context is the largest multiple of 8192 tokens that fits, at 8-bit / 4-bit KV, with 256 MiB spare and capped at 1M (GLM has no 4-bit KV):

† At --qwen-gdn 16 (bf16 GatedDeltaNet state), which the 32, 48 and 64 GB launch lines use. The f32 default gives less: 192k / 288k at 48 GB, and 968k / 1M and 520k / 824k at 64 GB.

A Mac with less memory than a Sushi pack can still serve it: --ssd-budget-gb N keeps N GiB resident and streams the routed experts from the SSD, with the same replies as a resident load, at a speed set by the SSD. A streamed load serves text unless --vision loads the vision tower, whose weights then come out of the N GiB.

Streaming on a 32 GB Mac

Sushi-2bpw streams on a 32 GB Mac with MTP on, at about 20 tok/s decode:

sudo sysctl iogpu.wired_limit_mb=27000
sushi serve --model ~/.sushi/models/Qwen3.8-Flash-Next-Sushi-2bpw --ssd-budget-gb 18 --ctx-size 66000 \
  --qwen-gdn 16 --expert-pick-tolerance 0.3

--qwen-gdn 16 stores Qwen's recurrent state in bf16 instead of the f32 default, which halves the state in every cache checkpoint (about 54 MB each) and is what the 32, 48 and 64 GB recipes run.

--ssd-budget-gb 18 keeps 18 GiB resident (trunk, KV cache and an expert cache) and reads the remaining experts from the SSD. --expert-pick-tolerance 0.3 is lossy: when a routed expert is not cached, it uses the best cached expert whose router probability is at least 0.7 of the missed one's, which saves an SSD read; leave it out for replies identical to a resident load. The plan may come within 1 GiB of the GPU limit (--wired-margin-gib N, 1 to 32). The GPU limit resets at reboot.

Benchmarks

Community reports, using llmprobe --bench-only:

License

MIT, for sushi and the mlx-serve code it forks (LICENSE); ported kernels and vendored code are listed in NOTICE. The Qwen packs follow the Qwen Community License and the MiMo pack Xiaomi's MIT license, stated on each Hugging Face page.