Mage-VL for MLX (Apple Silicon)

MLX port of microsoft/Mage-VL — a 5B image/video VLM (Qwen3-4B text backbone + from-scratch Mage-ViT "Codec-ViT" vision encoder) — built as a model plugin for mlx-vlm.

Results (M4, 16GB, 4-bit)

Runs end-to-end on Apple Silicon in 4-bit (~3.1GB weights, 5.3 bits/weight avg):

$ generate.py --image dog.jpg --prompt "Describe this image in detail..."
→ "A dog with a mix of black, white, and brown fur, sitting on a patterned rug,
   looking directly at the camera."          [decode 28.8 tok/s]

$ generate.py --video soccer-broadcast.mp4 --num-frames 8 --prompt "What is happening?"
→ "A post-match analysis of a football game" [prefill 4175 tok/31.9s, decode 11.2 tok/s]

Vision tower is numerically validated against the reference weights (max_abs_diff 3.0e-4, fp32, full 24 layers). Image preprocessing is bit-exact vs the HF processor.

Streaming event gate (streammind_gate)

Mage-VL's headline capability — proactive, event-gated streaming — is ported in mage_vl/streaming.py: PreNet → Mamba-1 SSM → PostNet → 4-layer Qwen3 producing a per-frame silent/speak score. Run the timeline over a video with scripts/stream.py (add --comment to have the decoder speak when the gate opens).

Validated (scripts/validate_gate.py): all 64 gate tensors load strictly, and the Mamba-1 mixer matches a canonical torch selective-scan reference to max_abs_diff 4.3e-7. (The reference gate uses CUDA-only mamba_ssm, so the mixer is validated against the math its kernel implements.) On the sample soccer clip (post-match analysis, no dramatic event) the gate correctly stays silent across all frames — routine content — while --comment still yields an accurate description.

Quantization sweep (M4, 16GB, scripts/benchmark.py)

4-bit is the recommended config for 16GB — ~1.6× faster decode and ~30% less memory (decode is memory-bandwidth bound, so smaller weights win). 8-bit produces noticeably richer output; pick it if you have RAM to spare. Convert either with scripts/convert.py --bits 4|8.

Status

Codec video: what "ported" means

The codec canvas generation (HEVC bit-cost readiness via the external cv-preinfer binary, or the DCVC-RT neural codec that shells out to the bundled neural_codec/ with checkpoints + CUDA) is a separate neural video codec — not reimplementable in numpy/MLX. mage_vl/codec_processing.py ports everything downstream of the engine and consumes a pre-generated codec asset dir. On Apple Silicon the practical video path is the frames backend (fully portable, default); use video_backend= "codec" with codec_dir= only when you have codec assets.

Architecture notes (from reading reference/modeling_mage_vl.py)

• Text uses plain 1D position ids (no 3D M-RoPE), so the stock qwen3 language model works unchanged. This is the big simplifier.
• Video == images at the model level. Mage-ViT is purely spatial; the processor aliases video frames as pixel_values/image_grid_thw and carries time via <X.X seconds> text tags. So image and video share one forward path.
• Vision encoder is SigLIP/Qwen2-VL-style: Conv2d patch-embed (kernel=stride=16, implemented as an equivalent Linear), pre-norm LayerNorm blocks, exact-GELU SigLIP MLP, 3D RoPE split 4:6:6 across (t,h,w) with interleaved rotate_half, block-diagonal attention over frame windows of 4, then a LayerNorm + 2-layer MLP merger that fuses 2×2 patch blocks down to the 2560-dim text space.

Memory (IMPORTANT — 16GB Apple Silicon)

The full model is 5B params (~10GB bf16 / ~20GB fp32). On a 16GB machine:

• Never instantiate the model in fp32. Convert straight to 4-bit (~2.7GB).
• Don't run a big download and a model at the same time.
• The HF golden dump (dump_reference.py) needs a full bf16 forward (>10GB peak); prefer running it on a ≥24GB machine, or accept swap.

Setup

uv venv .venv && uv pip install --python .venv/bin/python mlx mlx-lm mlx-vlm safetensors pillow numpy av
# register this package into mlx-vlm's model registry:
ln -sfn "$PWD/mage_vl" .venv/lib/python3.12/site-packages/mlx_vlm/models/mage_vl

Workflow

# 0. Structural test (safe, tiny, no weights)
python scripts/smoke_test.py

# 1. Get real weights
cd reference && git lfs pull --include="model-00001-of-00002.safetensors,model-00002-of-00002.safetensors" && cd ..

# 2. Convert -> MLX 4-bit  (memory-heavy; see note above)
python scripts/convert.py --hf-path reference --out mage-vl-mlx-4bit --bits 4

# 3. Golden dump from HF (memory-heavy; ideally on a bigger box)
python scripts/dump_reference.py --model reference --image reference/assets/<img> --out golden/case.npz --full

# 4. Validate MLX vs golden
python scripts/validate.py --mlx mage-vl-mlx-4bit --golden golden/case.npz

Run it (image path, after converting weights)

python scripts/validate_image_processor.py            # native vs HF preprocessing (no weights)
python scripts/generate.py --mlx mage-vl-mlx-4bit \
    --image reference/examples/dog.jpg --prompt "What animal is this?"
python scripts/generate.py --mlx mage-vl-mlx-4bit \
    --video reference/examples/soccer-broadcast.mp4 --num-frames 8 --prompt "What is happening?"

The native processor (mage_vl/image_processing.py, mage_vl/processing.py) is transformers-free: numpy/PIL preprocessing (bit-exact vs the HF slow processor), tokenizers for tokenization, jinja2 for the chat template.

Next

1. Numerical validation of the model forward (steps 3–4 above). Vision-feature max-abs-diff ~1e-2 in bf16 (looser for 4-bit); greedy argmax should match HF. This is the last thing gating a "known-correct" end-to-end run.
2. Quantization/perf sweep on M-series (tok/s, peak mem) + streaming (streammind_gate).
3. (Optional) wire the external codec engine for the codec video backend.

Layout

mage_vl/          # MLX model plugin (config, vision, mage_vl) + native processor
                  #   (image_processing, processing, video_processing, codec_processing)
scripts/          # convert, generate, benchmark, smoke_test, validate_*
reference/        # HF checkout (code + weights via git-lfs) — gitignored

License

Apache-2.0 (see LICENSE / NOTICE). This is a derivative inference port of microsoft/Mage-VL (Apache-2.0); it reuses the Qwen3 language model from mlx-vlm. Model weights are not included — they come from the upstream Apache-2.0 checkpoint and are converted locally.