MLX-Flash

Run AI models too large for your Mac's memory — at near-full speed.

70B on 32 GB. 200B+ on 48 GB. No extra quantization — uses the model's native precision.

The Problem MLX-Flash Solves

You have a Mac with 36 GB RAM. You want to run a good local model — say Qwen3-30B (needs ~18 GB).

Sounds like it fits, right? Except macOS uses 8-10 GB, your browser takes 3 GB, your IDE takes 2 GB. You're at 31 GB used before the model even loads.

What happens with Ollama / llama.cpp / MLX-LM: Your Mac starts swapping to SSD. Inference drops to 2-5 tok/s. Fans spin. The UI freezes. You force-quit and load a smaller model.

What happens with MLX-Flash: It reads macOS memory pressure in real-time, keeps the hot parts of the model in RAM, streams cold parts from SSD on demand, and runs at 80+ tok/s. No swap. No fan noise. Your browser and IDE keep working.

That's the entire product. Everything else supports this.

When does MLX-Flash actually help?

Be honest about when you need it and when you don't:

Real measured numbers

All on Apple M3 Max, 36 GB RAM, with a browser and VS Code open:

The 30B → 82 tok/s result is real and reproducible. The 8B result shows honesty: when the model fits, the difference is small.

How it works (one paragraph)

MLX-Flash predicts which parts of the model you'll need next (97% accuracy for MoE models) and keeps them in RAM. Everything else stays on SSD and streams in on demand. It reads macOS kernel memory stats (vm_statistics64) every inference call and auto-adapts — releasing cache when pressure rises, pre-fetching when there's headroom. For multi-user setups, a Rust proxy routes conversations to Python workers with session affinity so your KV cache stays warm.

Quick Start

Option A: pip (recommended)

pip install mlx-flash
mlx-flash-chat    # auto-selects best Gemma 4 model for your hardware

Option B: Homebrew (includes Rust sidecar)

brew tap szibis/mlx-flash
brew install mlx-flash
mlx-flash-chat

Option C: Docker (for CI/testing)

docker pull ghcr.io/szibis/mlx-flash:latest
docker run --rm ghcr.io/szibis/mlx-flash pytest   # run tests

Note: Docker runs tests and packaging only. For GPU inference, run natively on macOS with Apple Silicon.

Start the API server

# Works with LM Studio, Cursor, Claude Code, Codex, OpenAI SDK, and more
mlx-flash --port 8080

MLX-Flash auto-detects your hardware, picks the best Gemma 4 model for your RAM, and starts serving.

From source: git clone https://github.com/szibis/MLX-Flash.git && cd MLX-Flash && pip install -e ".[all]"

What MLX-Flash Actually Does Differently

Five things. Each one solves a real problem.

1. Runs models that don't fit in your RAM. Other tools crash or swap-thrash. MLX-Flash streams model parts from SSD and caches the hot ones in RAM. After ~25 tokens, 85-95% of accesses are served from RAM cache. A 70B model on a 32GB Mac runs at ~8 tok/s instead of not running at all.

2. Keeps your Mac usable while running large models. MLX-Flash reads macOS memory pressure in real-time (via kernel vm_statistics64, 0.1ms per check). When pressure rises — you open Chrome, Xcode, Slack — it shrinks its cache automatically. When pressure drops, it expands. Result: no beach balls, no frozen UI, no fan noise.

3. Multiple users on one machine. Rust proxy routes concurrent requests to N Python workers. Same conversation sticks to the same worker (KV cache stays warm). New conversations go to the least loaded worker. Three devs sharing a Mac Studio each get their own warm inference session.

4. Infinite-length conversations without running out of memory. StreamingLLM attention-sink KV eviction + quantized KV cache + attention-aware compression (H2O/ScissorHands). 4-bit KV gives 4x longer context at the same memory. Attention-aware eviction keeps only the tokens that matter — 5-20x KV compression with <1% quality loss.

5. Three speculative decoding strategies — pick the right one for your model. DFlash block diffusion for MoE models with trained drafters. LayerSkip self-speculative for dense models (no extra memory — the model IS the drafter). EAGLE-3 autoregression heads for hidden-state prediction. Plus Sequoia tree-structured speculation with SSD offloading for models that don't fit in RAM.

See docs/real-world-usage.md for 5 detailed scenarios with measured numbers, and docs/competitive-analysis.md for the full comparison.

How It Works

flowchart LR
    Client["Client<br/>(LM Studio, Cursor, SDK)"] --> Rust["Rust Sidecar<br/>:8080<br/>Session routing"]
    Rust -->|session sticky| W1["Python Worker 1<br/>:8081<br/>KV cache warm"]
    Rust -->|least loaded| W2["Python Worker 2<br/>:8082"]
    Rust -->|overflow| W3["Python Worker N<br/>:808N"]
    W1 --> Cache["Smart Cache<br/>97% hit rate"]
    Cache -->|hot path 0.08ms| RAM["RAM<br/>Hot experts"]
    Cache -->|cold path| SSD["SSD<br/>Full model"]
    RAM --> GPU["Metal GPU"]
    SSD --> GPU

Result: Models 2-5x larger than your RAM run at 2-3x faster than naive SSD streaming. After ~25 tokens, the cache learns your workload and hits 85-95% accuracy. Multiple workers bypass Python's GIL for concurrent request handling.

Supported Models

MLX-Flash works with any MLX-compatible model. It especially shines with large MoE (Mixture of Experts) models where only a fraction of parameters activate per token:

Get models from: HuggingFace mlx-community (MLX-native) | LM Studio (GUI download) | Ollama (ollama pull gemma4) | Kaggle (original weights)

Run mlx-flash-browse to see which models fit your specific hardware, or python -m mlx_flash_compress.hf_calculator to estimate memory for any model.

Performance

Real measured results — Apple M3 Max, 36GB RAM:

Qwen3-30B-A3B (MoE, 4-bit):       82.6 tok/s  ████████████████         30B model, only 2.1GB RAM free
Qwen1.5-MoE 14B (A2.7B, 4-bit):  122.1 tok/s  ████████████████████████ MoE, fits in RAM
Qwen3-8B (Dense, 4-bit):           53.5 tok/s  ██████████               Dense baseline
                                    ─────────
                                    30B MoE runs at 82 tok/s under memory pressure
                                    MoE is 2.3x faster than dense (only fraction of params active)

Memory pressure recovery — the key result:

Model at 0.9x RAM (barely fits):
  Without optimization:    43.5 tok/s  ########
  With MLX-Flash:         104.5 tok/s  ####################  2.4x faster

Cache warm-up — gets faster as it learns:

Token  0:  83.3ms (cold start)
Token  8:   5.7ms (warming up, 62% cache hit)
Token 24:   0.5ms (full speed, 85%+ hit)
         -> 41x speedup from warm-up

from mlx_flash_compress.expert_streaming import (
    enable_expert_streaming, enable_skip_fallback
)

streaming = enable_expert_streaming(model, capacity_per_layer=64)
enable_skip_fallback(model, streaming.caches, adaptive_skip_threshold=3.0)
streaming.warmup()

# For a 200GB model on a 48GB Mac
python -m mlx_flash_compress.tier_optimizer --total-ram 48 --model-gb 209

# Output: "Best: 41.5GB RAM cache, 82% of requests served from RAM -> 6.4 tok/s"

Using It

Multi-worker mode: Rust sidecar on :8080 routes to N Python workers on :8081-:808N. Same conversation sticks to the same worker (hot KV cache), new conversations go to the least loaded worker. All existing integrations work unchanged — clients still connect to :8080.

Chat commands: /models browse catalog, /model N switch live, /search web search, /ask search+answer, /remember save facts, /status memory info

Integrations

MLX-Flash connects to every major AI tool via three protocols:

pip install mlx-flash
mlx-flash --port 8080 --preload

{
  "mcpServers": {
    "mlx-flash": {
      "command": "python",
      "args": ["-m", "mlx_flash_compress.mcp_server"]
    }
  }
}

{
  "mcpServers": {
    "mlx-flash": {
      "command": "mlx-flash-server",
      "args": ["--mcp"]
    }
  }
}

mlx-flash --port 8080 --preload
export OPENAI_API_BASE=http://localhost:8080/v1
export OPENAI_API_KEY=not-needed

{
  "mcpServers": {
    "mlx-flash": {
      "command": "python",
      "args": ["-m", "mlx_flash_compress.mcp_server"]
    }
  }
}

mlx-flash --port 8080 --preload
export OPENAI_API_BASE=http://localhost:8080/v1
export OPENAI_API_KEY=not-needed
codex "refactor this function"

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
response = client.chat.completions.create(
    model="local",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)

mlx-flash --port 8080 --preload

# Ollama clients connect directly:
curl http://localhost:8080/api/generate -d '{"model":"local","prompt":"Hello"}'
curl http://localhost:8080/api/chat -d '{"model":"local","messages":[{"role":"user","content":"Hi"}]}'
curl http://localhost:8080/api/tags  # list loaded models

{
  "mcpServers": {
    "mlx-flash": {
      "command": "python",
      "args": ["-m", "mlx_flash_compress.mcp_server"]
    }
  }
}

Monitoring & Metrics

MLX-Flash exposes a Prometheus-compatible /metrics endpoint for production monitoring. Both the Rust proxy (:8080) and Python workers (:8081+) serve metrics.

# Quick check
curl -s http://localhost:8080/metrics | head -20

# Start Grafana + Prometheus (pre-configured, one command)
docker compose --profile monitoring up -d
open http://localhost:3000   # admin / mlxflash

Key metrics exposed:

Single scrape target: Prometheus only needs to scrape :8080/metrics — the Rust proxy aggregates all Python worker stats automatically.

Pre-built Grafana dashboard at dashboards/mlx-flash-overview.json — auto-provisioned with docker compose --profile monitoring up.

Structured logs — both Rust and Python emit unified structured logs (JSON or text) to stdout and optionally to file:

# JSON logs for Loki / Datadog / ELK
mlx-flash --port 8080 --log-format json

# JSON + file
mlx-flash --port 8080 --log-format json --log-file /var/log/mlx-flash.log

{"timestamp":"2026-04-06T14:32:01Z","level":"info","component":"python-worker","worker_port":8081,"message":"Model loaded","model":"Qwen3-30B","load_time_s":4.2}

See docs/metrics.md for the full metrics reference (30+ metrics), and docs/logging.md for structured logging, Vector/Loki config, and log field reference.

Web UI

MLX-Flash includes built-in web interfaces — no extra setup needed:

The dashboard and chat UI also work on standalone Python workers (:8081/admin, :8081/chat).

Worker management — control workers without restarting the server:

Workers are auto-health-checked every 10 seconds. If a worker dies, the Rust proxy automatically relaunches it.

Requirements

• macOS with Apple Silicon (M1/M2/M3/M4/M5)
• Python 3.10+
• 16 GB+ RAM (more = better caching = faster)

What's New

v0.7.0 — 11 New Features + Multi-Model Profiling

Speculative decoding (3 new strategies):

• LayerSkip — self-speculative decoding: model IS both draft and verifier, no extra memory needed. 1.8-2.2x on slow dense models. (arXiv:2404.16710)
• EAGLE-3 — lightweight autoregression head on hidden states. ~2x speedup, 8.5 MB draft head. Train in 1000 steps. (EAGLE paper)
• Sequoia — tree-structured speculation with SSD offloading. Draft in RAM, verify from SSD. 5-10x offload speedup. (arXiv:2402.12374)

KV cache optimization (3 new features):

• StreamingLLM — attention-sink KV eviction for infinite-length generation with fixed memory. Keep first K "sink" tokens + sliding window. (arXiv:2309.17453)
• Quantized KV cache — 4-bit keys/values = 4x longer context at same memory. 2/4/8-bit modes with per-group absmax quantization.
• ScissorHands/H2O KV compression — attention-score-based eviction. Keep only "heavy hitter" tokens + recent window. 5-20x KV compression. (arXiv:2306.14048)

MoE acceleration (2 new features):

• Dynamic expert pruning — skip low-weight experts (gate weight <5% of top-1). 15-30% fewer FLOPs on MoE models. Adaptive threshold.
• Shared expert pinning — detect DeepSeek/Qwen "always-on" experts, pin in hot cache, never evict. 10-20% fewer SSD reads.

Dense model optimization (2 new features):

• Layer-wise quantization — first/last layers at Q8 (sensitive), middle at Q3/Q4 (robust). Extends mixed-precision from MoE experts to dense layers.
• MatFormer elastic inference — extract sub-models from 50-100% of full size. Auto-shrink on memory pressure, restore when headroom returns. (arXiv:2310.07707)

Server (1 new feature):

• Continuous batching — vLLM-style dynamic batching with chunked prefill, KV cache pooling, and thread-local MLX streams. 2-4x server throughput.

Infrastructure:

• Multi-model profiling — scripts/bench_multi_profile.py auto-profiles all models, recommends DFlash config
• Central model registry — scripts/models.yaml with cleanup (--cleanup), listing (--list), RAM filtering
• 805 tests (was 355), all passing
• Worker pool — run N Python workers behind Rust sidecar (--workers 3)
• Session-sticky routing — same conversation stays on same worker (hot KV cache)

v0.6.1 — Multi-Precision + Performance

• 7-tier quantization — FP16/Q8/Q6/Q5/Q4/Q3/Q2 auto-assigned by expert activation frequency
• 23% less memory on MoE models with realistic power-law expert distribution
• 30% more experts cached → higher hit rate → faster inference
• 320 tests, 91% coverage

v0.6.0 — Gemma 4 + Engineering

• Gemma 4 as default model — auto-detects best model for your RAM
• Page cache control — madvise(MADV_FREE) keeps memory pressure 20% lower
• Pipelined execution — phase-level IO/compute overlap (15-25% faster)
• Metal kernels — fused Q4 dequant+GEMV, SwiGLU, MoE dispatch
• Bit-parity verified — FP32 accumulation proves 0.0 delta from streaming
• mlx-lm patch — transparent Flash mode for LM Studio

See the CHANGELOG for the full history.

Complete Feature Stack (38 Features)

Every optimization in MLX-Flash, grouped by what it does. All measured numbers are from real hardware.

Auto-Profiling & Profiles

MLX-Flash auto-detects your model and picks the optimal configuration. No manual tuning needed.

from mlx_flash_compress import profile_and_configure

model_info, profile = profile_and_configure(model, tokenizer)
# Automatically selects: quality, speed, balanced, or skip

Decision logic:

Manual override:

# Prioritize quality (bf16 drafter, max acceptance)
model_info, profile = profile_and_configure(model, tokenizer, priority="quality")

# Prioritize speed (8-bit drafter, smaller blocks)
model_info, profile = profile_and_configure(model, tokenizer, priority="speed")

# Let auto-detect decide (default)
model_info, profile = profile_and_configure(model, tokenizer, priority="auto")

Profile all your models at once:

python scripts/bench_multi_profile.py          # profile all registered models
python scripts/bench_multi_profile.py --list   # show cached models + sizes
python scripts/bench_multi_profile.py --cleanup # remove all cached models

Roadmap

What's next, in priority order. Each item has a published paper with measured results.

Next up (v0.7.1):

• LayerSkip cached inference path — current implementation re-encodes full context; need incremental KV cache for draft phase to realize the 1.8-2.2x speedup on slow dense models
• Expert pruning native integration — move from monkey-patching to native MoE forward pass to eliminate patching overhead
• EAGLE-3 pre-trained heads — publish trained draft heads for Gemma 4 31B, Qwen3.5-27B, Devstral 24B
• DFlash model-specific drafters — train and benchmark on the three recommended models (15-21 tok/s range)

Medium-term (v0.8.0):

• Sequoia end-to-end — wire SSD layer offloading with tree speculation for 70B on 32GB at 30-50 tok/s
• StreamingLLM + quantized KV integration — connect to actual model attention layers, not just the cache abstraction
• KV compression online — wire H2O/ScissorHands attention tracking into live inference
• Continuous batching integration with serve.py — merge with existing server for production multi-user

Longer-term (v1.0.0):

• Custom Metal attention kernels — sliding window, paged attention for KV cache management
• JACCL distributed inference — multi-Mac cluster via Thunderbolt 5
• Structured output / grammar-guided generation — eliminate JSON/code parsing failures
• MatFormer-trained checkpoints — publish models with nested FFN structure for elastic extraction

See docs/honest-benchmarks.md for the full feature expansion roadmap with measured impact numbers.

Deep Dive

License

MIT