cellm — Mobile-Native LLM Serving Engine

A ground-up LLM inference engine for iOS and Android, written in Rust. Brings server-grade serving concepts — paged KV cache, continuous decode scheduling, multi-session concurrency — to phones running under 512MB RAM.

Not a wrapper around llama.cpp. Not a port of vLLM. A new runtime designed for mobile constraints from scratch.

[!NOTE] Status. The CPU and Metal backends, the iOS bindings, and the text models listed under Supported Models are stable and in active use. Metal coverage varies by model; CPU runs everything. The Android AAR and Compose demo build and package the native library, but haven't been validated or tuned on-device. Vulkan sets up the device and pipelines but still runs every op on CPU. The native .cellm VLM path and the Docker image are experimental or not yet wired up; vision via ONNX is stable. See Feature Status.

Quick Links

Quick Start

Prerequisites

• Rust 1.75+ (modern stable toolchain)
• macOS / iOS for Metal acceleration (Linux/Android builds use CPU path)
• Git LFS (for bundled sample models)

1. Build

cargo build --release

2. Run a smoke test (CPU)

cargo run --release --bin infer -- \
  --model models/smollm2-135m.cellm \
  --tokenizer models/hf/smollm2-135m/tokenizer.json \
  --prompt "Hello, how are you?" \
  --chat \
  --gen 32

3. Run with Metal (macOS/iOS)

cargo run --release --bin infer -- \
  --model models/smollm2-135m-int8.cellm \
  --tokenizer models/hf/smollm2-135m/tokenizer.json \
  --prompt "Hello" \
  --chat \
  --gen 16 \
  --backend metal

Tip: Use --chat for ChatML-style formatting. Without it, many base models behave like text-completion engines and may not answer directly.

4. Metal verification

cargo run --release --bin metal-smoke

---

Architecture Overview

flowchart LR
    U["User Prompt"] --> API["App/UI Request Layer"]
    API --> ORCH["CPU Orchestrator"]
    ORCH --> TOK[Tokenizer]
    TOK --> FMT["Prompt Formatter"]
    FMT --> SCH["Decode Scheduler / Batcher"]

    SCH -->|"prefill/decode jobs"| ENG["Engine Dispatcher"]
    ENG -->|backend=CPU| CPUPATH["CPU Kernels"]
    ENG -->|backend=Metal| METAL["Metal Kernels"]

    METAL --> MATMUL["QKV / MLP MatMul"]
    METAL --> ATTN["Attention + GroupKV Cache"]
    METAL --> NORM["RMSNorm / RoPE / Logits"]
    MATMUL --> SAMPLER
    ATTN --> SAMPLER
    NORM --> SAMPLER
    CPUPATH --> SAMPLER["Sampler + Stop Rules"]

    SAMPLER --> DETOK[Detokenizer]
    DETOK --> STREAM["Streaming Output"]
    STREAM --> API
    API --> U

    subgraph ModelAssets["Local Model Assets"]
      W[".cellm / .cellmd mmap Weights"]
      T["tokenizer.json + config"]
    end

    W --> ENG
    T --> TOK

    subgraph SessionState["Per-Session State"]
      KV["KV Cache (GroupKV layout)"]
      PT["Page Table / Sequence Cursor"]
      TH["Thermal + QoS Policy"]
    end

    SCH --> KV
    SCH --> PT
    ORCH --> TH
    TH --> SCH

---

What Makes cellm Different?

---

Project Structure

cellm/
├── crates/
│   ├── cellm-core/          # Memory arena, tensor layout, op dispatch
│   ├── cellm-model/         # Model format, configuration, weight management
│   ├── cellm-cache/         # Paged KV cache: BlockAllocator, PageTable, physical storage
│   ├── cellm-kernels/       # CPU, Metal, WASM & WebGPU compute kernels
│   ├── cellm-scheduler/     # Decode scheduler & batching logic
│   ├── cellm-wasm/          # WebAssembly bindings & JavaScript API
│   └── cellm-sdk/           # Public C FFI + high-level API for mobile consumers
├── bindings/
│   ├── ios/CellmDemo/       # SwiftUI demo app (LLM + VLM stub)
│   ├── kotlin/              # Android Kotlin/JNI bindings
│   └── swift/               # Swift Package + XCFramework build scripts
├── tools/
│   ├── infer/               # CLI inference runner (debug & validation)
│   ├── vlm-onnx-infer/      # VLM runner for SmolVLM ONNX exports
│   ├── vlm-smoke/           # SDK FFI VLM smoke test
│   ├── convert/             # HF Safetensors/GGUF/PyTorch -> .cellm converter
│   ├── bench/               # Latency & throughput benchmark harness
│   └── metal-smoke/         # Minimal Metal kernel compile + dispatch test
├── docs/                    # Architecture deep-dives, benchmarks, model guides
└── models/                  # Sample .cellm checkpoints (Git LFS)

---

Development Commands

Convert a Model

Convert HuggingFace Safetensors or GGUF to .cellm:

cargo run --bin convert -- \
  --input  ./models/hf/smollm2-135m \
  --output ./models/smollm2-135m.cellm \
  --dtype  f16

Quantize during conversion:

cargo run --bin convert -- \
  --input  ./models/hf/smollm2-135m \
  --output ./models/smollm2-135m-int8.cellm \
  --dtype  f16 \
  --quantize-int8-symmetric

See docs/convert-quantized-models.md for GGUF, PyTorch, and 4-bit affine workflows.

Run Benchmarks

# Quick smoke benchmark
cargo run --release --bin bench -- --model tiny

# Full LLM backend matrix (CPU vs Metal)
tools/bench/run_llm_backend_matrix.sh

Detailed benchmark reports live in docs/benchmarks/.

Run VLM (Vision-Language)

# ONNX vision + ONNX decoder (recommended)
cargo build --release -p cellm-vlm-onnx-infer

./target/release/vlm-infer \
  --model-dir models/hf/smolvlm-256m-instruct \
  --onnx-variant fp16 \
  --image models/test_images/rococo.jpg \
  --prompt "Describe this image." \
  --split-image \
  --max-new-tokens 96

Native .cellm vision + decoder is experimental:

./target/release/vlm-infer \
  --model-dir models/hf/smolvlm-256m-instruct \
  --cellm-model models/smolvlm-256m.cellm \
  --vision-backend cellm \
  --decoder-backend cellm \
  --image models/test_images/rococo.jpg \
  --prompt "Describe this image." \
  --max-new-tokens 12

See docs/vlm-smolvlm-onnx.md for full VLM docs.

iOS SwiftUI Demo

Build the XCFramework:

./scripts/build_xcframework.sh

Then open bindings/ios/CellmDemo in Xcode.

Browser / WebAssembly

Build and run the WASM engine with WebGPU acceleration:

# Build the WASM module
./scripts/build-wasm.sh --release

# Serve the demo page
python3 -m http.server 8080 --directory crates/cellm-wasm/www/

Then open http://localhost:8080 and use engine.try_init_webgpu() to enable hardware acceleration.

---

Docker

cellm's whole point is phone deployment (iOS/Android via Metal/Vulkan), so Docker can't run the actual mobile targets — there's no GPU passthrough for Apple's Metal API in a Linux container. Where Docker does help is the CPU-backend and tooling side of the repo: CI, headless benchmarking, model conversion pipelines, and testing without owning a Mac.

Unlike llama.cpp's Docker matrix (CUDA/ROCm/Vulkan/SYCL variants for full/light/server), cellm doesn't need a wide backend matrix — this is a mobile-first inference engine, not a server deployment target. A single CPU image covers the actual need for reproducible tooling.

Example Dockerfile shape:

FROM rust:1.75 AS build
WORKDIR /app
COPY . .
RUN cargo build --release --bin infer --bin convert --bin bench

FROM debian:bookworm-slim AS cpu
COPY --from=build /app/target/release/infer /app/target/release/convert /app/target/release/bench /usr/local/bin/
ENTRYPOINT ["infer"]

Usage once built:

docker run -v /path/to/models:/models cellm:cpu \
  --model /models/smollm2-135m.cellm \
  --tokenizer /models/hf/smollm2-135m/tokenizer.json \
  --prompt "Hello" --chat --gen 32

Not wired up yet — tracked as a follow-up. If you want to help, a Dockerfile + GitHub Actions workflow (mirroring .github/workflows/docker.yml-style multi-arch builds) for the cpu variant is the right first PR.

---

Git Commit Messages

Use cellm's local LLM to generate git commit messages from staged changes. All inference runs locally on CPU — no API keys, no data leaves your machine.

Prerequisites

• A model converted to .cellm format (e.g., Qwen2.5 0.5B int8)
• The matching tokenizer.json
• ./target/release/infer built (cargo build --release)

Quick usage

# Stage your changes first
git add -A

# Generate a commit message from staged diff
./tools/git-cellm-commit.sh

# Or pipe any diff
 git diff HEAD~1 | ./tools/git-cellm-commit.sh

How it works

1. git diff --cached captures your staged changes
2. The diff is passed as the prompt to Qwen2.5 0.5B int8
3. The model generates a commit message + summary locally
4. Output is printed to stdout — pipe it anywhere

Configuration

Set these environment variables to use a different model:

Example with the LFM model:

CELLM_COMMIT_MODEL=models/LFM2.5-230M-int4-v2.cellm \
CELLM_COMMIT_TOKENIZER=models/to-huggingface/LFM2.5-230M/tokenizer.json \
  ./tools/git-cellm-commit.sh

---

Supported Models

Recommended first download: SmolLM2-135M

Sample checkpoints bundled in this repo (via Git LFS):

• models/smollm2-135m-int8.cellm
• models/smolvlm-256m-int8.cellm
• models/qwen3.5-0.8b-int4-textonly.cellm

---

Feature Status

• Paged KV Cache - Fixed-size block allocation with BlockAllocator & PageTable
• Multi-session Scheduler - Round-robin interleaved decoding
• 4-bit Affine Dequantization - Native MLX/HF packed weight support
• Multimodal Vision (ONNX) - SmolVLM via ONNX runtime; stable, see docs/vlm-smolvlm-onnx.md
• Native .cellm VLM path - ViT/SigLIP encoder + linear projector (experimental, see docs/vlm-smolvlm-onnx.md)
• Accelerated Math - Metal + WASM SIMD + WebGPU compute kernels
• WebAssembly Support - Run LLMs in the browser with wasm-bindgen
• High-Performance CLI - Conversion, benchmarking, debug inference
• Git Commit Messages - Generate commit messages from local LLM via tools/git-cellm-commit.sh
• Vulkan Support - Device/pipeline/buffer management is in place, but every compute op still falls back to CPU (research)
• Android Integration - Kotlin/JNI bindings, AAR, and Compose demo app build against a cross-compiled .so; on-device validation and tuning still open
• Qwen iOS Porting - Optimize Qwen inference for native iOS
• Docker (CPU tooling image) - Reproducible CI/benchmarking image for infer/convert/bench (see Docker)

---

Documentation Index

---

Troubleshooting

Metal is not being used

# 1. Verify Metal device access
cargo run --release --bin metal-smoke

# 2. Verify infer picks Metal
./target/release/infer \
  --model models/smollm2-135m-int8.cellm \
  --tokenizer models/hf/smollm2-135m/tokenizer.json \
  --prompt "hello" --gen 8 --backend metal

In restricted/sandboxed shells, Metal device discovery can fail. infer --backend metal now errors instead of silently falling back to CPU.

SmolLM2 360M needs non-interleaved RoPE

CELLM_LLAMA_ROPE_INTERLEAVED=0 ./target/release/infer ...

Gemma-3 Metal quality knobs

Default keeps norm/RoPE/logits on CPU-safe path for quality parity. Opt-in Metal paths:

CELLM_GEMMA_USE_METAL_NORM=1   # enable Metal RMSNorm
CELLM_GEMMA_USE_METAL_ROPE=1   # enable Metal RoPE
CELLM_GEMMA_USE_METAL_LOGITS=1 # enable Metal final logits matvec

Llama graph path (experimental speed)

CELLM_LLAMA_ENABLE_GRAPH=1 ./target/release/infer ...

Model-specific env flags

For more debug flags and backend-specific notes, see the per-model docs in docs/.

---

Citation

If you use cellm in your research, please cite:

@software{asante2026cellm,
  author       = {Asante, Jeffrey},
  title        = {cellm: A Mobile-Native LLM Serving Engine},
  year         = {2026},
  url          = {https://github.com/jeffasante/cellm},
  note         = {Rust inference engine with paged KV cache and multi-session scheduling}
}

---

License

Apache License, Version 2.0 (LICENSE-APACHE)