QW (QuasarWave)

QW is a focused, local LLM inference runtime for single-user Apple Silicon systems. It serves the Qwen3.8 27B dense checkpoint through an OpenAI-compatible HTTP API and includes a command-line client for downloading models, generating responses, and inspecting the local cache.

QW is intentionally narrow: MLX is the supported backend, and the default model is Jundot/Qwen3.8-27B-oQ4e-fp16-mtp. It is not a general multi-model or CUDA/ROCm runtime.

QW also supports:

• MTP with included heads in the default model checkpoint
• DFlash2 speculative decoding (depends on --features dflash2, enabled by default)
• Image vision
• Grammar constrained decoding

Showcase

Asciicast demo implementing a QR Code generator webapp(no speedup):

The Pi agent with QW built this QR-code app in 2m 40s:

Install

Prebuilt binaries can be installed with the release installer:

curl --proto '=https' --tlsv1.2 -LsSf https://github.com/cr0sh/qw/releases/download/v0.1.0/qw-cli-installer.sh | sh

To compile from source, use macOS on Apple Silicon with the Rust toolchain, CMake 3.16 or newer, and the Xcode Command Line Tools:

xcode-select --install
cargo install --locked --git https://github.com/cr0sh/qw qw-cli

To install a tagged release, add --tag <tag> to the cargo install command.

Quickstart

Download the default target checkpoint and the optional DFlash2 draft checkpoint:

qw download
qw download incoai/Qwen3.8-27B-DFlash2

Start the local server:

qw serve

The server listens at http://127.0.0.1:8000 by default. Its OpenAI-compatible base URL is http://127.0.0.1:8000/v1; supported POST endpoints include /chat/completions and /responses.

For example:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"qwen3.8-27b","messages":[{"role":"user","content":"Hello!"}],"max_tokens":64}'

Generate one response without starting the HTTP server:

qw generate --prompt 'Explain prefix caching in one sentence.'

Inspect local model and cache state, or view all command-line options:

qw stats
qw --help
qw serve --help
qw generate --help

Models and decoder options

The target model can be selected by checkpoint path with --model or QW_MODEL_PATH; when neither is set, QW resolves the default model from its cache. The optional DFlash2 draft checkpoint can be selected with --dflash-draft-model or QW_DFLASH_DRAFT_MODEL_PATH.

Both qw generate and qw serve expose --decoder auto|baseline|mtp|dflash. The dflash option requires a build with the dflash2 feature and an available draft checkpoint. Use --help for the options enabled by the current build.

Historical performance

Latest complete cargo bench results (tokens/s), with the implementation at commit 7894f5d:

Fresh prefill processes 4,341 prompt tokens; cached-prefix prefill processes only the 288 newly appended tokens, not the cached prefix. Decode excludes prefill and measures 127 output tokens after the first token.

Earlier bundled-MTP (k=3) decode results at commit 117fecb were 57.354, 52.639, and 34.853 tokens/s for fresh, 10k, and 64k respectively; these are historical measurements, not a same-revision decoder comparison.

Both runs used the default Jundot/Qwen3.8-27B-oQ4e-fp16-mtp checkpoint with Turbo4 KV cache on a Mac Studio with an Apple M4 Max 40-core GPU and 64 GB of unified memory.

Development

Cache layout, GPU serialization, and benchmark commands are collected in DEVELOPMENT.md. When running GPU work from more than one worktree, always invoke it through ./gpu-lock -- ...; do not delete the shared lock file.

Evaluation

Qwen3.8-27B (medium reasoning) achieved 43.3% Pass@1 on τ³-Banking (42/97 tasks, one trial per task), compared with 47.4% reported by Artificial Analysis as an average over five runs. Our single-run result may be affected by sampling variability. The evaluation took approximately 26 hours on a single M4 Max machine, so we did not run additional trials.

The τ³ banking evaluation launch and capture rules are documented in eval/README.md.

Acknowledgements

This repository is heavily AI-assisted. Commit messages include an Assisted-by footer identifying the coding agent/model used.

QW is inspired by antirez's ds4 inference engine for its minimalism and simplicity.

The repository started from a stripped version of mlxcel-core from mlxcel. We appreciate Lablup's open-source contributions.

QW adopts optimization approaches from oMLX, ds4, and qwen-3.8-mtp-challenge. Derivative work is attributed in the relevant commit messages.