vllm-ios

vLLM-style continuous batching for iPhone. Native Swift on MLX, no Python.

vllm-ios is an embeddable inference engine focused on one thing: the fastest multi-agent LLM serving on iOS. If your app runs several model calls concurrently — agent fleets, parallel RAG extraction, fan-out research — this engine batches them through one set of weights so every forward pass and every byte of memory bandwidth does maximum work.

On an iPhone, that matters more than anywhere else: you get roughly 15–20 seconds of full-speed GPU per minute before thermal throttling takes over. Whatever inference you're doing needs to be over as fast as possible. Background and origin story: Continuous Batching on an iPhone.

Unaffiliated with the vLLM project. Architecturally inspired by vLLM and vllm-mlx; shares no code with either.

SwarmBench: one question, one batched decode, eight agents streaming. Video in the blog post.

Measured on an iPhone 16 Pro

Qwen3.5-0.8B, 4-bit, greedy decoding, thermally controlled runs:

At 8-bit, against llama.cpp with functionally identical weights (Q8_0 vs MLX 8-bit), level for level:

End to end: 16 research requests (~17k prompt tokens, 2k generated tokens of structured JSON) in 25 seconds, inside one thermal budget, with sub-second admission of new requests into a running batch (best measured TTFT under load: 0.94s).

Scheduling overhead is measurably zero: per-stream throughput inside the full engine matches a hand-rolled static batch exactly. All numbers come from thermally gated runs (cooling gaps between levels; an iPhone throttles below what ProcessInfo.thermalState reports) — raw result data is in docs/benchmark_results_20260825.json.

Across everything measured, quantization and batching were the two levers that mattered — both attack memory bandwidth, and they multiply: 8-bit single-stream (51 tok/s) → 4-bit batch-8 (199 tok/s) is a 3.9x swing on the same silicon. Everything else tuned (flash attention, KV layouts, chunk sizes) moved results by single-digit percentages.

How it works

The engine is ~300 lines on top of mlx-swift-lm, using stock MLX kernels and public API only. Three ideas:

1. Uniform-offset continuous batching. Every sequence in the running batch always shares the same KV offset, so the model's stock causal-mask path works unchanged — no ragged attention masks, no model forks. A request arriving mid-flight is left-padded with filler tokens to the group's current offset, prefilled solo, then spliced in. For workloads with similar-length prompts the padding waste is ~1%.
2. Cache surgery. Join and exit happen at token boundaries by concatenating or index-selecting each layer's KV tensors along the batch axis — via KVCache.state accessors and ArraysCache.extend/filter, so it works across cache types, including the recurrent caches of hybrid models like Qwen3.5 (GatedDeltaNet).
3. Chained greedy decode. With greedy sampling the token-feedback loop never needs the CPU: several decode steps are built into one lazy graph (argmax feeds the next step on-GPU) and evaluated with a single sync per chunk.

Prefill runs in chunks that evaluate only cache state, so lm_head output for prompt positions is never computed — without this, batched prefill of 8×1k-token prompts materializes a ~2.6 GB logits tensor and iOS jetsam kills the app.

Usage

import VLLMiOS
import MLXLMCommon

// context: a ModelContext from mlx-swift-lm (any LLM it can load)
let engine = VLLMEngine(
    model: context.model,
    padToken: 198,          // a harmless filler token, e.g. "\n"
    maxBatch: 4             // 2–4 is the sweet spot on A-series
)

let requests = prompts.enumerated().map { i, tokens in
    EngineRequest(id: i, promptTokens: tokens, maxTokens: 128,
                  arrivalTime: CFAbsoluteTimeGetCurrent())
}
let report = try engine.run(requests: requests)
for result in report.results {
    print(result.id, result.ttft, result.tokens.count)
}

Streaming: pass onTokens to receive each request's tokens as they materialize — once at first token, then every decode chunk (4–8 tokens, ~150–300ms apart at phone speeds). Set engine.decodeChunk = 1 for strict per-token streaming at a small throughput cost (the chunked lazy graph is worth ~7%):

let report = try engine.run(requests: requests) { id, newTokens, done in
    // called on the engine's thread; hop to your actor before touching UI
    render(id, detokenizer.append(newTokens), finished: done)
}

Prefix caching: agent fleets share a system prompt, and prefill dominates agent workloads. Freeze the shared prefix once and every request prefills only its suffix:

let shared = PrefixCache.longestCommonPrefix(of: prompts)   // token-level LCP
let prefix = try engine.cachePrefix(shared)                  // prefilled once
let report = try engine.run(requests: requests, prefix: prefix)

Measured (M-series Mac, 16-request burst, 174-token shared prefix): wall 8.4s → 7.6s and TTFT p50 −11%, with the prefix snapshot built once in 0.09s. Correctness note: the transplanted KV state is exact, but suffix prefill uses different chunk boundaries than full prefill, so greedy outputs can differ at rare numeric near-ties (15/16 byte-identical in testing) — the same caveat vLLM documents for its prefix caching.

On iOS, cap MLX's buffer cache or the recycled-buffer pool will count against the jetsam limit:

MLX.GPU.set(cacheLimit: 64 * 1024 * 1024)

The vllm-bench executable target reproduces the benchmark scenarios (burst and staggered-arrival) on macOS:

# Metal shaders require the Xcode build system (plain `swift build` runs
# CPU-only): build once via xcodebuild, then run the product.
xcodebuild -scheme vllm-ios -destination 'platform=macOS' \
  -configuration Release -derivedDataPath .xcbuild \
  -skipMacroValidation -skipPackagePluginValidation build
./.xcbuild/Build/Products/Release/vllm-bench --n 16 --batch 8 --scenario both

What this is and isn't

Is: an embeddable continuous-batching engine — request queue, arrival-time admission, batched prefill, lockstep batched decode, early exit. Quant-agnostic (runs whatever mlx-swift-lm loads, including calibrated dynamic quants like OptiQ).

Isn't (yet): an OpenAI-compatible server, a streaming token API, a prefix cache, a sampler (greedy only), or a ragged-batch scheduler. Roadmap, in value order:

• Prefix caching (see below)
• Streaming token callbacks (chunk-granularity; decodeChunk = 1 for strict per-token)
• Sampling beyond greedy
• Per-sequence RoPE offsets + padding masks (drop the uniform-offset restriction; mlx-swift-lm already ships half the plumbing)

SwarmBench (demo app)

SwarmBench/ is the demo app: a chat UI where each message fans out to 1–8 specialist agents (key facts, plan, risks, contrarian, …) served concurrently by the engine — batched decode, a per-turn shared-prefix cache, streaming tokens into a live card grid, and early exit as each agent finishes. In a typical clip: 8 agents, 385 tokens, 2.9 seconds, on-device. The header has a model selector that switches between the four quants with live reload, an agent-count control, and chat history; settings manage on-device model storage. Open SwarmBench/SwarmBench.xcodeproj, set your team, run on a device; models download on first use.

Related work

License

Apache-2.0