mlxcel

High-performance LLM, VLM, embedding, reranking, and audio inference for Apple Silicon and NVIDIA CUDA systems, with experimental AMD ROCm support on Linux. The CLI and server are implemented in Rust and execute MLX SafeTensors checkpoints through native MLX C++ bindings, without Python in the request path or a checkpoint-conversion step.

Overview

mlxcel is both a local inference CLI and a production-oriented model server. It runs text generation, multimodal input, embeddings, reranking, speech workloads, continuous batching, prompt caching, speculative decoding, and distributed inference in one native runtime.

Apple Silicon is the primary target. Linux/CUDA is supported as a secondary target. AMD GPUs on Linux (ROCm) are an experimental, source-build-only target, and an opt-in OpenXLA/IREE backend is available as an alpha development path.

Current main highlights

The current main branch is v0.7.0 plus unreleased work. Install from source to use features that have not reached the latest tagged release; see the changelog for the release-by-release record.

• Experimental AMD GPU (ROCm) support. --features rocm builds mlxcel on Linux against an MLX ROCm backend vendored into the source tree as the ROCm part of mlxcelverse, applied on the same pinned MLX commit as the Metal and CUDA builds. Affine, mxfp8, and mxfp4 checkpoints, including gpt-oss MoE experts, run on RDNA 3.5 (gfx1151); fused kernels fall back to MLX graphs and affine MoE models need MLXCEL_FUSED_MOE=0 for now. See Linux with AMD ROCm for the build and the open gaps.
• Verified llama-server compatibility. A frozen b10621 manifest classifies all 376 pinned options, routes, and native request fields, with no deferred entries. Native completion, embedding, tokenization, template, infill, props, slots, metrics, resumable-stream, router, and LoRA surfaces are implemented or explicitly classified instead of being silently ignored.
• OpenAI, Anthropic, and Vertex-compatible serving. Chat Completions, Completions, Responses, Embeddings, Reranking, Audio, and Anthropic Messages are available alongside the native llama-server routes. Optional Vertex AI custom-container routing is supported through the standard AIP_* variables.
• Multi-model routing and live adapters. Router mode discovers checkpoints from the model store, a model directory, or INI presets; loads them on demand; and bounds the resident set with LRU eviction. Multiple LoRA adapters can remain unfused for per-request or live scale changes, or be fused for zero decode overhead.
• Broader multimodal and retrieval coverage. Embedding and reranker families include BERT, ModernBERT, SigLIP, Qwen3/Qwen3-VL, Llama/Nemotron, LFM2.5, ColBERT-style models, cross-encoders, and generative rerankers. Qwen-VL video input and Responses-native image parts are supported.
• Expanded audio serving. The compatible transcription boundary recognizes WAV, MP3, and FLAC by content, and chat-model transcription streams one ASR delta per decoded token. Phi-4 Multimodal and Gemma 3n decode all three containers; other current audio families and dedicated Whisper remain WAV-only. Kokoro provides text-to-speech.
• Measured speculative decoding. Gemma 4, Qwen and GLM-4.7-Flash MTP paths, the LFM2 / LFM2.5 DSpark path, and the Muse Glimmer assistant path, probe greedy exactness before enabling the fast path, and the server exposes the adaptive decision at GET /v1/internal/mtp-policy. LFM2.5 targets pair with LiquidAI's published DSpark drafters on mlxcel-server (--draft-model), greedy decoding only. Muse Glimmer pairs with meta-models/Muse-Glimmer-30B-assistant on text-only requests, and the round loop measures its verify width instead of fixing it. mlxcel split-mtp extracts the GLM-4.7-Flash drafter from the raw checkpoint.
• Operational controls. Optional authenticated live settings, prompt/cache and slot observability, bounded response stores, idle model sleep/wake, GBNF grammars, expanded sampling controls, runtime reasoning placement, and API-key/CORS/TLS controls are available.
• DeepSeek-V4 support. The deepseek_v4 architecture includes HyperConnections, rotating shared-KV attention, per-layer compression, HiSA sparse selection, and hash-routed early MoE layers.
• Inkling across four modalities. The text backbone runs hybrid sliding/global NoPE attention with per-layer short-convolution state and logsigmoid-normalized experts, and carries HMLP image tiling, adjacent-frame video, dMel audio, and a native MTP drafter.
• Half-precision paths keep their dtype. Activation helpers, the multimodal rotary, the Gemma3n load policy, and the GraniteMoeHybrid gated norm no longer widen a half-precision hidden state to f32, which had made every later matmul promote its own weight. The affected checkpoints move by up to 3.26x, and the bridge's reductions accumulate in f32 to close a bfloat16 max that returned NaN for a finite input.

See Server features for the route and deployment map and llama-server compatibility for the exact b10621 boundary.

Why mlxcel

• Native, compact runtime surface. Loading, scheduling, and inference stay in one process, with no Python environment or interpreter layer to provision in production.
• Direct checkpoint loading. MLX SafeTensors checkpoints from HuggingFace, including mlx-community, load directly. Many standard SafeTensors embedding checkpoints also load without conversion.
• Simple deployment artifacts. mlxcel and mlxcel-server are native executables suitable for packaging and service supervision. Platform runtime libraries are still required.
• Compatibility without overclaiming. mlxcel-server accepts a broad llama-server flag and LLAMA_ARG_* environment surface, while the checked-in manifest records every supported, aliased, not-applicable, or intentional-difference case.
• Serving features enabled for real workloads. Continuous batching, prompt-prefix caching, and automatic prefix caching are on by default. Speculative decoding, KV-cache compression, router mode, live LoRA, and distributed modes are available where the model and backend support them.
• Reproducible model surgery. Default builds support YAML load-time weight edits through --surgery / MLXCEL_SURGERY, including scale, add, prune, replace, and interpolate.
• Broad architecture coverage. Dense transformers, sparse MoE, hybrid SSM, VLM/OCR, block diffusion, embeddings, rerankers, ASR, and TTS are represented. Run mlxcel arch for the binary's architecture catalog and consult Supported models for checkpoint-level notes.

Quick start

Install with Homebrew

The Homebrew formula installs the latest released mlxcel and mlxcel-server binaries on macOS and Linux:

brew tap lablup/tap
brew install mlxcel

Run a model

mlxcel run resolves the model, downloads it on first use, reuses it afterward, and starts an interactive chat REPL. A bare model name resolves under mlx-community by default.

# Interactive chat.
mlxcel run Qwen3.5-0.8B-4bit

# One-shot generation, then exit.
mlxcel run Qwen3.5-0.8B-4bit -p "Hello, world!" -n 100

# With no model argument, use mlx-community/gemma-4-e2b-it-4bit.
mlxcel run

generate, serve, and inspect accept the same model forms through -m: an existing local path, a HuggingFace owner/name repository id, or a bare name. MLXCEL_DEFAULT_ORG changes the bare-name organization.

Thinking-capable checkpoints may generate reasoning before the final answer. run and generate hide reasoning from terminal output by default; pass --show-reasoning to display it without the raw marker tokens.

With no -n/--max-tokens, generation continues until EOS or the effective context limit. Pass -n N to set a cap.

mlxcel generate -m Qwen3.5-0.8B-4bit -p "Hello, world!" -n 100

# Read-only memory estimate before loading.
mlxcel inspect -m Qwen3.5-0.8B-4bit --max-tokens 32768

# Emit the same estimate as machine-readable bytes for recipe builders or schedulers.
mlxcel inspect --json -m Qwen3.5-0.8B-4bit --max-tokens 32768 | python3 -m json.tool

# Abort before generation if the estimated model and KV cache do not fit.
mlxcel generate -m Qwen3.5-0.8B-4bit -p "Hello" -n 32768 --estimate-memory

mlxcel inspect --json prints a single JSON object with byte-exact weights_bytes, kv_bytes_total, activation_bytes, headroom_bytes, budget_bytes, total_bytes, fits, input flags, and per-token FP16/INT8 KV rates when the model config exposes KV geometry. TurboQuant per-token sizing is reported as null until the estimator models those widths directly.

Start a server

mlxcel-server and mlxcel serve start the same server. Their server flag surfaces are kept aligned; mlxcel serve additionally has the one-shot preflight flags --estimate-memory and --force.

mlxcel-server -m Qwen3.5-0.8B-4bit --port 8080

curl http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen3.5-0.8B-4bit","messages":[{"role":"user","content":"Hello"}]}'

Common deployment controls work on both server entry points:

# API keys from a list or file; both sources are combined.
mlxcel-server -m Qwen3.5-0.8B-4bit --api-key alice-key,bob-key
mlxcel-server -m Qwen3.5-0.8B-4bit --api-key-file /etc/mlxcel/api-keys

# Restrict browser CORS, or serve HTTPS.
mlxcel-server -m Qwen3.5-0.8B-4bit \
  --allowed-origins https://app.example.com,https://admin.example.com
mlxcel-server -m Qwen3.5-0.8B-4bit \
  --ssl-cert-file cert.pem --ssl-key-file key.pem

# Serve below a path prefix or on a Unix domain socket.
mlxcel-server -m Qwen3.5-0.8B-4bit --api-prefix /llama
mlxcel-server -m Qwen3.5-0.8B-4bit --host /run/mlxcel.sock

Downloaded models normally live at ${MLXCEL_CACHE_DIR:-$HOME/.cache/mlxcel}/models/<owner>/<name>. On generate, run, inspect, download, list, and rm, use --models-dir to override that store. On the two server entry points, the store override is --model-store-root; --models-dir now selects multi-model router discovery and cannot be combined with -m.

# Router mode: discover direct checkpoint subdirectories under /srv/models,
# use the global store as another source, and keep at most four models loaded.
mlxcel-server --models-dir /srv/models \
  --model-store-root /var/lib/mlxcel/models --models-max 4

Router management, live LoRA, native routes, runtime settings, context sizing, idle sleep, and compatibility migration details are in Server features.

Embed and rerank

The offline commands use the same loaders and batching paths as the HTTP endpoints.

# Embed two texts and print vectors plus cosine similarity.
mlxcel embed -m sentence-transformers/all-MiniLM-L6-v2 \
  -p "The weather is lovely" -p "It is sunny"

# Rank documents against one query.
mlxcel rerank -m BAAI/bge-reranker-v2-m3 \
  -q "what is panda?" -d "hi" -d "The giant panda is a bear species."

# Serve embeddings alone, or a reranker beside a chat model.
mlxcel-server -m sentence-transformers/all-MiniLM-L6-v2 --port 8080
mlxcel-server -m Qwen3.5-0.8B-4bit \
  --reranker-model mlx-community/Qwen3-Reranker-0.6B-4bit --port 8080

See Embeddings and reranking for multimodal inputs, supported families, pooling, request schemas, and side-model serving.

Transcribe and synthesize audio

An audio-capable chat checkpoint serves transcription. Phi-4 Multimodal and Gemma 3n accept WAV, MP3, and FLAC detected from their bytes; other current audio families accept WAV. A dedicated Whisper checkpoint also provides a WAV-only transcription server.

mlxcel-server -m models/whisper-base --port 8080

curl http://localhost:8080/v1/audio/transcriptions \
  -F file=@recording.wav -F model=whisper-base

Kokoro-82M checkpoints serve POST /v1/audio/speech. See Audio API for model layout, request formats, limits, streaming, and current backend constraints.

Manage downloaded models

mlxcel list                  # name, size, and modified time
mlxcel list --sort size      # largest first
mlxcel list --json           # stable machine-readable array
mlxcel list -q               # repository ids only
mlxcel list -v               # include absolute paths

mlxcel rm mlx-community/Qwen3.5-0.8B-4bit
mlxcel rm mlx-community/Qwen3.5-0.8B-4bit --yes

mlxcel rm deletes only from the mlxcel-managed store. A checkpoint that exists solely in the read-only HuggingFace cache is reported but not removed.

Other CLI tools

# Object detection with an RT-DETRv2 checkpoint.
mlxcel detect -m models/rt-detr-v2 -i image.jpg --format json

# Inspect the hardware-specific kernel tuning matrix without profiling it.
mlxcel tune --dry-run

# Split the GLM-4.7-Flash MTP block out of the raw checkpoint into a 4-bit
# drafter, then pair it with a glm4_moe_lite target.
mlxcel split-mtp -m models/glm-4.7-flash-bf16 -o models/glm-4.7-flash-mtp-4bit --q-bits 4
mlxcel generate -m models/glm-4.7-flash-4bit --draft-model models/glm-4.7-flash-mtp-4bit -p "Hello"

mlxcel tune can profile supported kernel tactics into the local autotune cache; set MLXCEL_AUTOTUNE=cache to consume recorded winners. Use mlxcel --help or a subcommand's --help for the complete flag surface.

Build from source

Apple Silicon prerequisites are Rust, Xcode Command Line Tools, CMake, and the Metal toolchain component:

xcodebuild -downloadComponent MetalToolchain
git clone https://github.com/lablup/mlxcel.git
cd mlxcel
cargo build --release --features metal,accelerate

Linux/NVIDIA builds require the CUDA toolkit and MLX's CUDA system dependencies:

cargo build --release --features cuda

Linux/AMD builds are experimental and need a ROCm installation with its HIP, rocBLAS, rocWMMA, hipBLASLt, and hipRTC CMake packages:

cargo build --release --features rocm

A plain Linux build has no GPU feature and runs on the CPU, which is not a validated release target and is much slower. See Installation for the complete prerequisite, CUDA and ROCm architecture, runtime-header, and packaging matrix, and Linux with AMD ROCm for the current ROCm status.

Performance

Measured on the commits released as v0.7.0 against same-host, same-day runs of mlx-lm 0.31.3 and mlx-vlm 0.6.17. Text is the 2026-09-06 sweeps at a fixed pp512/tg128 shape; VLM is the 2026-09-09 sweeps, where both the runtime and the reference were re-measured in one pass; qwen3-vl-reranker-2b is excluded from the VLM rows because a reranker driven through a generation harness is not measuring the VLM path. These are medians over a roster, not guarantees for an individual checkpoint.

Two of these numbers replace larger ones and the reason is the measurement condition rather than a regression. The 2026-05-19 campaign on mlxcel 0.0.28 reported text prefill at 2.78x on M5 Max and 1.79x on M1 Ultra. Those ran on a short natural prompt whose token count differed between the two runtimes, and prefill_tok_s is prompt tokens divided by prefill time, so a difference in tokenization lands directly in the ratio. At a synthetic 512-token prompt that both runtimes receive identically, prefill is close to parity. The older figures should not be quoted for the current release.

Named checkpoints, M5 Max, mlxcel against the same-day reference. Text rows are mlx-lm 0.31.3, VLM rows are mlx-vlm 0.6.17:

The split in that table is architectural rather than arbitrary. Across the whole M5 Max text roster, mixture-of-experts checkpoints run at a median 106% of mlx-lm (19 pairs, quartiles 100 and 108) while dense ones sit at 100% (69 pairs, quartiles 99 and 100). Every checkpoint above 105% in the table is MoE, and every dense entry is at parity, including qwen3.8-27b-4bit, mixtral-8x7b and the dense Gemma 4 QAT variants. Gemma 4 QAT splits along the same line rather than along quantization: the 26B A4B mixture reads 108% on text and 113% on VLM, while the dense E2B, E4B and 31B QAT checkpoints read 100%, 100% and 99%. Read a dense model's row as "no reason to switch runtimes for speed" and an MoE model's as the case where mlxcel's expert dispatch is worth something.

The VLM figures also moved between mlxcel releases, and unevenly. Re-sweeping that table at one commit left the median row unchanged at 1.00x against its previous reading, with 9 of 71 rows moving more than 10%: mistral-small-4-119b-2603-4bit 5.30x, moondream2 4.19x, qwen3-vl-30b-a3b 2.61x, qwen3-omni-30b-a3b 2.57x, then five between 1.16x and 1.46x. The gains come from dtype fixes that reach specific paths, so the aggregate moved from 98% to 105% while most individual rows stayed where they were. Read the per-model tables rather than the median if a particular checkpoint matters.

Per-host sweeps are more current than the combined report. M5 Max and M1 Ultra are both measured on the v0.7.0 commits (2026-09-06 through 2026-09-09) at the same MLX pin and the same pp512/tg128 shape, so cross-host ratios between those two are taken on one version. The per-host records label those runs 0.7.0-beta.1, which is the crate version string the tree carried until the release commit. Coverage differs by host rather than version: M5 Max carries text, VLM, speculative decoding, batched serving and the embedding and rerank ladder, while M1 Ultra carries text, VLM, speculative decoding and embeddings. GB10 is still on v0.4.0-rc.1 and the older measurement shape, so any ratio involving it mixes versions. Focused reports cover paged attention, MoE, KV compression, embeddings, and speculative decoding. Read Benchmark results, the benchmark report, and the methodology before comparing hosts or planning capacity.

MTP performance depends on the target/drafter pair, prompt, verify width, and hardware generation. With the v0.6.0 exactness policy, Gemma 4 12B plus its 4-bit assistant measured 93.2 tok/s against 43.8 tok/s classic decode on M5 Max (2.13x, greedy output byte-identical). The faster inexact kernel measured 120.4 tok/s but is not the default. How much that is worth depends heavily on the prompt: on the same host and release, the speculative_bench --sweep default 14-token prompt measures the same pairing at 70.9 tok/s against a 45.3 tok/s baseline (1.57x at K=4, 35.0% acceptance), against the 2.13x above on a code prompt. The runtime measures exactness and profitability instead of treating either speedup as universal; see Speculative-decoding acceptance, MTP policy API, and the benchmark records under docs/benchmark_results/.

Re-run the supplied harnesses on the target checkpoint, prompt shape, context length, and hardware before treating any published number as a deployment expectation.

Supported models

Model support is architecture- and checkpoint-dependent. Run:

mlxcel arch
mlxcel arch --json

for the architecture catalog compiled into the current binary. The --json form emits the stable recipes registry snapshot used by downstream site builds and automation, including standalone runtime families such as detector-only checkpoints. Supported models maintains the family table, quantization support, distributed coverage, checkpoint qualifications, and known caveats.

Python

The pure-Python mlxcel client manages a local mlxcel serve process or connects to an existing server, discovers the served model id, and exposes the underlying OpenAI client for the full API surface.

import mlxcel

with mlxcel.LLM("mlx-community/Qwen3-4B-4bit") as llm:
    print(llm.generate("def fib(n):", max_tokens=128))
    for delta in llm.stream("Write a haiku about autumn"):
        print(delta, end="", flush=True)

Install with pip install ./python. See Python client for managed and connect modes, async use, streaming, structured output, and troubleshooting.

Optional GUI

mlxcel-server can be used directly through HTTP clients. Backend.AI Go is available as a companion UI for local chat, model management, and multi-model routing.

Documentation

• Server features and route map
• llama-server b10621 compatibility boundary
• Installation and build prerequisites
• Environment variables
• Supported models
• Architecture overview
• Continuous batching and disaggregated serving
• Tensor and pipeline parallelism
• OpenAI Responses API
• Embeddings and reranking
• Audio API
• TurboQuant KV cache
• Speculative-decoding acceptance
• Adaptive MTP policy API
• Benchmarks and methodology
• Python client
• Adding a new model

The documentation index lists design notes, benchmark records, and specialized implementation guides that do not need to live in this README.

Contributing

Issues and pull requests are welcome. See CONTRIBUTING.md for the contributor workflow, local quality gates, and commit conventions. For larger changes, open an issue first so the scope and validation plan can be discussed.

For security vulnerabilities, see SECURITY.md; do not file them as public issues.

License

Apache License 2.0 unless otherwise noted. See LICENSE. Third-party attributions carried forward under Apache-2.0 Section 4(d) are listed in NOTICE.

Acknowledgments

• MLX, Apple's machine learning framework.
• NripeshN/mlx, whose rocm-support branch is the AMD GPU (ROCm/HIP) MLX backend vendored as the ROCm part of mlxcelverse, under the MIT License. See NOTICE.
• mlx-lm, mlx-vlm, and mlx-audio, whose model coverage and behavior mlxcel ports and mirrors. See NOTICE.
• MLX Community, pre-converted MLX checkpoints.
• turboquant_plus, whose TurboQuant KV-cache algorithms are ported under Apache-2.0. See NOTICE.
• moonshotai/Kimi-K3, whose tiktoken pre-tokenization pattern and XTML chat grammar are reimplemented in Rust under the Kimi K3 License. See NOTICE.
• FlashInfer, whose paged-attention, split-KV, cascade state-merge, and sampling designs inform mlxcel's serving kernels.
• IREE and StableHLO, the runtime and graph format behind the alpha OpenXLA backend: mlxcel emits StableHLO, iree-compile lowers it, and the IREE runtime executes it under the opt-in xla-iree build. See NOTICE.