Metal Marlin - Fast LLM Inference on Apple Silicon

GLM-4.7-Flash at 35 tok/s on M4 Max • OpenAI-Compatible API • Production Ready

Quick Start • Serving • Performance • Docs

---

Performance

GLM-4.7-Flash (35 tok/s on M4 Max)

• Throughput: 35.2 tok/s decode
• Latency: 28.6 ms/step
• Memory: 12.4 GB
• Optimization: 4.9× speedup from baseline

See optimization details →

Requirements

• macOS 13.0+ (Ventura or later)
• Apple Silicon (M1/M2/M3/M4, M4 Max recommended)
• Unified Memory: 32GB+ recommended
• Python 3.12 (via uv)

Install

git clone https://github.com/RESMP-DEV/metal-marlin.git
cd metal-marlin
uv sync --extra all

Quick Start

GLM-4.7-Flash Server (Production)

Start an OpenAI-compatible server at 35 tok/s:

# Start server with optimized config
uv run python scripts/serve_glm47.py \
  --model-path ./models/glm47-flash-mmfp4 \
  --port 8000 \
  --max-batch-size 32

# Test with any OpenAI SDK
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"glm-4.7-flash","messages":[{"role":"user","content":"Hello"}],"max_tokens":50}'

Features:

• ✅ OpenAI API compatible (drop-in replacement)
• ✅ Request batching (32 concurrent)
• ✅ PagedAttention KV cache
• ✅ Streaming support
• ✅ Metrics endpoint

Full Guide: docs/QUICKSTART_SERVING.md

Python API

from metal_marlin.inference.mmfp4_pipeline import MMFP4Pipeline
from metal_marlin.trellis.config import GLM4_TOKENIZER_ID

# Load optimized pipeline (35 tok/s)
pipeline = MMFP4Pipeline.from_pretrained(
    "./models/glm47-flash-mmfp4",
    device="mps"
)

# Generate
output = pipeline(
    "Explain quantum computing",
    max_new_tokens=100,
    temperature=0.7
)
print(output)

Validation

Run end-to-end validation:

# Validate TPS (target: 35 tok/s) and perplexity
./tests/validation/run_all_validation.sh

See Validation Checklist for details.

---

Serving

OpenAI-Compatible Server

Start server:

uv run python scripts/serve_glm47.py \
  --model-path ./models/glm47-flash-mmfp4 \
  --host 0.0.0.0 \
  --port 8000

Endpoints:

OpenAI SDK Compatible:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")

response = client.chat.completions.create(
    model="qwen3_4b_fp4",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True
)

for chunk in response:
    print(chunk.choices[0].delta.content or "", end="")

Quantization

Convert models to efficient quantized formats:

# Basic FP4 quantization
metal-marlin quantize Qwen/Qwen3-4B --format fp4 -o qwen3_4b_fp4

# Trellis v2 (recommended for MoE models)
metal-marlin quantize Qwen/Qwen3-30B-A3B --format trellis-v2 -o qwen3_30b_trellis

# Qwen3.5-122B-A10B MMFP4 (CUDA MR-GPTQ)
uv run python scripts/quantize_qwen35_122b_a10b_mmfp4_cuda.py \
  --output models/Qwen3.5-122B-A10B-MMFP4

For MoE models, dynamic bit allocation reduces size 40-50% with <1% quality loss. See Quantization Guide for details.

C++ Extension (Optional, 5-10x Faster Dispatch)

mkdir -p build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j
cp _cpp_ext.cpython-312-darwin.so ../metal_marlin/

Check availability:

from metal_marlin import fast_dispatch_available
print(f"Fast dispatch: {fast_dispatch_available()}")  # True if C++ ext built

Documentation

Development

uv run pytest tests/ -v               # Full test suite
uv run pytest tests/ -v -m smoke      # Quick smoke tests
uv run ruff check .                   # Linting
uv run pyright metal_marlin/          # Type checking

License

Apache 2.0