ZeroHop

Speculative decoding across the Apple Neural Engine and GPU — on one consumer chip, via public APIs only.

A 1B draft model runs on the ANE (Core ML), an 8B target verifies on the GPU (MLX/Metal), and a measurement harness characterizes the seam between them. Apple M3 · 16 GB · macOS 26.5.1 · July 2026.

Results

• The ANE draft is fully hidden inside the GPU's verify window: overlap-span p50 = 150.465 ms vs verify p50 = 150.463 ms — a 2 µs difference, holding across the whole k-sweep (up to 9 ANE calls per batch). Drafting costs the GPU nothing.
• Output is token-exact with the baseline in every greedy cell; the sampling mode (Leviathan rejection acceptance) is distribution-exact by construction.
• The ANE→GPU handoff is p99 165 µs (10,000-iteration protocol) once two power-state effects, neither in Apple's documentation, are controlled — three orders of magnitude below the compute terms.
• The measured viability arithmetic, all constants from this hardware: T_draft(k=4) = 95.6 ms + T_handoff(p99) = 0.15 ms < T_verify = 144.4 ms (34% margin).

Boundary conditions

• Against a fast 3B target speculation loses everywhere (same-GPU 0.58×, ANE serial 0.45×) — the win regime is the memory-bound large-target regime the theory predicts.
• k has a knee at 6: at k=8, acceptance decay (78% → 73%) and verify-window growth outpace tokens/round.
• Sampling halves acceptance (T=0.8: 44.6% ANE / 29.9% GPU draft) and serial speculation goes unprofitable — the known temperature sensitivity of speculation. The fp16-activation ANE draft agrees with the target's distribution notably better than the end-to-end 4-bit GPU draft in both regimes.

Harness findings

The project ran measurement-first: a synchronization harness with an explicit kill criterion (p99 handoff > 2 ms ⇒ stop) ran before any model existed. Everything is reported as p50/p95/p99/p99.9 over ≥10,000 iterations per cell — means are never a success criterion. Highlights:

• The dominant tail effect is the GPU idle-clock ramp — not ANE cold-start. The effect is known to latency-sensitive Metal developers (Anukari's waste makes haste, Apple forums) but absent from Apple's documentation, and as far as I can find this is its first measurement as a tail distribution. Uncontrolled: p99 1.33 ms, p99.9 6.29 ms, max 22.9 ms on the event-signal→kernel-start segment. A saturated GPU collapses it to p99 165 µs; trickle keep-warm does not remove the deep tail; the surviving ~1-in-10k events are run-to-run ambient (OS preemption, not power state — established by a repeat run).
• The ANE warmth axis is flat at a 20 Hz dispatch duty cycle: the workload's own traffic is heartbeat enough. (A 1k-iteration run had shown a "cold event" that vanished at 10k — one of two conclusions the full protocol reversed.)
• Both zero-copy ANE→GPU transport paths are real (IOSurface→MTLTexture, and page-aligned makeBuffer(bytesNoCopy:)), verified by per-iteration backing-identity asserts; statistically indistinguishable at 10k. A coherency canary — computed by the ANE's own matmul, validated by the GPU kernel before reading the payload — has zero stale reads in 26,000+ iterations.
• Public-API dispatch overhead: a minimal ANE dispatch round trip costs ~300 µs p50 through release-build Core ML vs the ~95 µs private-path figure from maderix's benchmarking (this bounds the overhead; it doesn't isolate it). Core ML's cost model refuses per-token-scale ANE dispatches (multi-token drafting is required, not just faster), and the ANE compiler rejects compiled weights above ~1 GB (2.3 GB fp16 → wholesale CPU; 591 MB 4-bit palettized → preferred=ane — a cliff independently reported by Benazir & Lin and the reason ANEMLL chunks models at ~950 MB).
• 

The ANE deployment recipe

Getting a real stateful-KV-cache Llama-3.2-1B onto the ANE through public API hit four walls; each was isolated with a discriminating experiment (Tools/convert_draft.py is the recipe as executable code):

Wall 4 was invisible to output correctness (the target corrects every wrong draft — only the acceptance rate betrayed it); a torch-vs-Core ML greedy A/B is now a mandatory draft-health assertion. A side effect of the fix: speculation rollback is O(1) — move the position pointer, re-mask, no cache surgery.

Prior art: the static-shape rule and the fixed-window approach were published by Panaro (Oct 2024), and stateful pure-ANE inference was already shipping in ANEMLL. What I could not find documented anywhere: wall 1's all-or-nothing granularity (one dynamic slice bound rejects the entire program — the coremltools FAQ implies per-layer fallback), and wall 4's silent failure signature plus the one-hot fix — Apple's own stateful-models guide publishes the vulnerable cache-write pattern unwarned.

Landscape (July 2026)

No published work I could find holds all four of: (1) two-model speculation, (2) ANE-draft + GPU-verify on one consumer chip, (3) public API only, (4) rigorous handoff characterization. Every neighbor holds at most two:

Notes:

• Mirror-SD is the concept prior art: Apple's own research maps speculation across heterogeneous accelerators (GPU + NPU) at server scale (2.8–5.8×, 14B–66B) — but names no consumer device, imposes no public-API constraint, and characterizes no handoff. ZeroHop is a consumer-silicon, public-API implementation of that design class, with the handoff measured.
• The gap is probably not oversight. Prior ANE work (maderix, Orion, ane.cpp, ANEForge) uses private APIs, citing Core ML's scheduler overhead. This project measures what the public path — the only one App Store software can ship — actually costs.
• Attribution correction: the widely-quoted ~0.095 ms ANE dispatch figure originates from maderix's private-API benchmarking (credited by Orion) — it is not an Orion measurement.

On a single engine: ane.cpp's self-speculation on the ANE is slower than plain decoding (draft and verify serialize on one engine — the same contention as the same-GPU control above); ANEForge's two-model speculation on the ANE alone reaches 2.28×. ZeroHop runs the draft on an engine that is otherwise idle.

Repository map

The measurement binary (zerohop) has zero third-party dependencies; MLX is confined to the separate end-to-end runner (zerohop-m3).

Reproducing

Requirements: Apple Silicon Mac (macOS 15+; developed on M3/26.5.1), Xcode command-line tools, Python 3.12 for model conversion only.

swift build -c release                          # measurement harness (no deps)
python3.12 -m venv Tools/.venv && Tools/.venv/bin/pip install coremltools numpy matplotlib
Tools/.venv/bin/python Tools/make_models.py Models
xcrun coremlcompiler compile Models/m1_tiny.mlpackage Models/

.build/release/zerohop m0                                                     # noise floor
.build/release/zerohop m1 --e3 sync --e4 rt --e5 none --gpu-warm saturated    # handoff
.build/release/zerohop m2 --e1 A --gpu-warm saturated                         # transport + canary

# end-to-end (downloads models; xcodebuild required — SwiftPM can't compile MLX's Metal shaders)
Tools/.venv/bin/pip install "torch==2.7.0" "transformers==4.53.3"
Tools/.venv/bin/python Tools/convert_draft.py Models --nbits 4
xcrun coremlcompiler compile Models/draft_llama32_1b.mlpackage Models/
xcodebuild build -scheme zerohop-m3 -configuration Release -destination 'platform=macOS' \
  -derivedDataPath .build/xcode -skipPackagePluginValidation
.build/xcode/Build/Products/Release/zerohop-m3 --k 6 --n 200 \
  --target mlx-community/Meta-Llama-3.1-8B-Instruct-4bit \
  --coreml-draft Models/draft_llama32_1b.mlmodelc --overlap on

Protocol: ≥500 warmup + ≥10,000 measured iterations per cell, release builds only (debug builds inflate the dispatch segment ~3×), plugged in, thermal-gated. Every run exports meta.json (chip, OS build, power source, thermal timeline) + per-iteration samples.csv to Results/.

Limitations

One chip (findings are expected to vary by ANE generation — raw data ships for replication); greedy is the optimized regime; the ANE draft is per-call latency-bound (~14–20 ms/token; a trained Medusa-class multi-token head would cut this, and its gains convert to speculation-window headroom rather than device contention); energy unmeasured (powermetrics needs root); the public-vs-private tax number bounds rather than isolates.

References

• Leviathan, Kalman, Matias. Fast Inference from Transformers via Speculative Decoding. ICML 2023. arXiv:2211.17192
• Chen et al. Accelerating LLM Decoding with Speculative Sampling. arXiv:2302.01318
• Apple ML Research. Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference. arXiv:2510.13161
• Orion: Characterizing and Programming Apple's Neural Engine for LLM Training and Inference. arXiv:2603.06728
• Cai et al. Medusa. arXiv:2401.10774 · Miao et al. SpecInfer. arXiv:2305.09781
• Lu et al. DuoDecoding. arXiv:2503.00784 · Dovetail. arXiv:2412.18934
• Apple ML Research. Recurrent Drafter (ReDrafter). arXiv:2403.09919
• ANEForge · ane.cpp · ANEMLL · CoreML-LLM · MLX / mlx-swift-lm
• SqueezeBits. Disaggregated Inference on Apple Silicon: NPU Prefill and GPU Decode. blog
• vllm-mlx arXiv:2601.19139 · Apple-Silicon runtime comparison arXiv:2511.05502
• Bryngelson. Apple Neural Engine: Architecture, Programming, and Performance. arXiv:2606.22283 · Benazir, Lin. arXiv:2604.18788
• maderix. Inside the M4 Apple Neural Engine. Part 1 · Part 2
• Anukari devlog. Waste makes haste…? · Panaro. Fast KV-Cached Attention for the ANE

Landscape statements are as of July 2026 (prior-art notes updated August 2026).