Core AI is Apple's on-device ML runtime in iOS 27 / macOS 27 and the successor to Core ML: PyTorch models are exported with Apple's coreai-torch (LLMs: coreai.llm.export) into .aimodel bundles that run on the GPU or the Neural Engine, e.g. Qwen3-8B 4-bit decodes at 94 tok/s on an M4 Max GPU, MLX 90 under the same protocol (apple-silicon-llm-bench, macOS 27 beta, 2026-06).

MiniCPM-V-4.6 — Core AI

On-device vision-language model for iPhone / Apple Silicon. A Core AI port of openbmb/MiniCPM-V-4.6 — the strongest sub-2B open VLM — running fully local on the GPU via the Core AI pipelined engine: pick a photo, ask about it, stream the answer.

Verified on iPhone 17 Pro: image → grounded answer at ~51.5 tok/s decode, all local.

Fridge photo → recipe ideas, fully on-device on an iPhone 17 Pro (CoreAIChat).

Use it

⚡ One line — run the kit's task op on this model (import CoreAIOps; no session, no model plumbing, downloads on first use):

let caption = try await CoreAI.caption(imageAt: url, options: .model("minicpm-v-4.6"))

Every op, one shape — Cookbook.

▶️ Run it (source) — the VLChat runner (GUI + CLI, one app for every vision-language model in the catalog):

git clone https://github.com/john-rocky/coreai-kit
open coreai-kit/Examples/VLChat/VLChat.xcodeproj
# → Run, then pick "MiniCPM-V 4.6" in the model picker

# agents / headless (macOS):
cd coreai-kit/Examples/VLChat
swift run vlchat-cli --model minicpm-v-4.6 --image sample.jpg --prompt "What is in this image?"

💻 Build with it — complete; the glue is kit API, copy-paste runs:

import CoreAIKit
import FoundationModels

let vlm = try await KitVisionModel(catalog: "minicpm-v-4.6")
let session = LanguageModelSession(model: vlm)
let image = try ImageFile.load(imageURL)  // any image file → CGImage + EXIF orientation
let reply = try await session.respond(to: Prompt {
    prompt
    Attachment(image.cgImage, orientation: image.orientation)
})
// reply.content: the answer about the image, generated fully on-device

The take-home is Examples/VLChat/Sources/QuickStart.swift — this exact code as one typed function, no UI; the CLI is an argument shell over it, and the GUI drives the same KitVisionModel(catalog:) behind a LanguageModelSession. Multi-turn about the same image? Hold the LanguageModelSession and call respond(to:) per turn. The photo picker / file chooser is your app's own chrome — ImageFile.load (kit API) turns any image file into model input.

Integration checklist

• SPM: https://github.com/john-rocky/coreai-kit → product CoreAIKit
• Info.plist: NSPhotoLibraryUsageDescription — only if you use PhotosPicker
• Entitlements (iOS): com.apple.developer.kernel.increased-memory-limit
• First run downloads the model — 2.1 GB (Mac) / 2.1 GB (iPhone) — then it loads from the local cache (Application Support; progress via the downloadProgress callback)
• Measure in Release — Debug is ~3× slower on per-token host work

Architecture

MiniCPM-V-4.6 (1.3B) = a SigLIP So400m vision tower (980px / patch 14 / 27 layers, with a window-attention insert-merger @ layer 6 + a downsample-MLP merger → ÷16 = 64 visual tokens per 448px slice) + a Qwen3.5-hybrid text backbone (qwen3_5_text: 0.8B, 24 layers, GatedDeltaNet linear attention ×3 : full attention ×1, head_dim 256, vocab 248094, tied head). Connector = 2×2 spatial merges + MLP, spliced into the text embeddings at <image> positions (masked_scatter).

Bundles

Recommended (optimized, 2026-06-25):

The int8 head quantizes the big-vocab LM head (fp16 in int8lin = ~half the per-token read) → +48% decode on iPhone 17 Pro (46→68 tok/s). The int8 vision halves the encoder's size (the encode is compute-bound, so this is a size/memory win); pair it with a one-shot vision-graph warmup at load to hide the ~2.7 s first-photo cold compile. Original …_int8lin decoder + fp16 minicpmv46_vision remain for compatibility.

The decoder is a complete qwen3.5-hybrid text LLM when image_embeds is zero — same bundle, no image needed.

How a VLM rides the text-only engine

The pipelined engine knows nothing about images. The whole multimodal state rides the static-input hook (image_embeds buffer) + an id-space trick — the graph stays ids + positions → logits:

• The host runs the vision encoder once per image (resize 448, normalize x/127.5−1) and writes image_embeds [64,1024] into one owned MTLBuffer the engine binds on every step.
• The prompt's <|image_pad|> ids are rewritten to extension ids V + slot (slot 0..63). In-graph: embed = ids < V ? table[ids] : image_embeds[ids − V].
• Positions are plain 1D (no M-RoPE / no rope-shift), the qwen3.5-hybrid KV + conv + recurrent states are the engine's; nothing else changes.

Simpler than the Qwen3-VL port (no deepstack, no M-RoPE).

Measured (iPhone 17 Pro, iOS 27 beta, release)

• iPhone 17 Pro decode (int8 head): VLM in-app A/B, same conditions, back-to-back — int8lin 51.5 → int8hu 70.0 tok/s = +36% (the VLM bundle binds image_embeds every step, which dilutes the head gain; the text core alone is 46 → 68 = +48%). ~64–70 tok/s in practice by device temperature. · M4 Max text core ~224 tok/s (llm-benchmark), engine cold-spec ~2–4 s, ~1.5 GB resident (jetsam-safe).
• Vision: int8 encoder ≈ fp16 encode time (compute-bound) at ~0.6 GB (half); the ~2.7 s first-image latency is the SigLIP graph's cold compile — run a dummy encode at load to make the user's first photo warm (~tens of ms).
• Numerics: fp32-torch parity bit-exact (vision cos 1.000000, full overlay logits cos 1.00004); Core AI engine ≡ python ≡ HF (text 24/24; image path reproduces the HF description modulo one int8 near-tie token, then reconverges).
• Real-photo example (kakigōri): "a bowl of shaved ice ... chunks of mango ... a dark blue saucer ... a menu or a book, hinting at a café ... a wooden table" — accurate, fully on-device.

Use it

apps/CoreAIChat and the standalone MiniCPMVLM app have a MiniCPM-V 4.6 mode with a photo picker: pick an image, ask, stream. The vision tower runs once per image (~hundreds of ms); each turn re-prefills (S=1).

Conversion + gates: see coreai-model-zoo / minicpm-v-4.6.

License: Apache-2.0 (inherited from openbmb/MiniCPM-V-4.6).

---

More models in this format: Core AI Model Zoo — 75 models, each with the recipe that produced it.

Want a different model on-device? Open a request — free, open weights only; the export and its measured numbers get published publicly.