Curated list of papers, frameworks, benchmarks, and applications for multimodal AI agents (LLMs, text-to-image, speech, world models, etc.) on mobile and edge devices.
39
30 commits
updated Sep 23, 2026
README
Awesome Edge AI for Multimodal Agents 
A curated list of papers, frameworks, benchmarks, and applications for efficient multimodal agents (LLMs, text-to-image, speech, world models, etc.) on mobile and edge devices.
Focused on inference engines, optimization, and deployment for real-world use.
📑 Contents
- Introduction
- Papers
- On-Device Models
- Frameworks & Inference Engines
- Hardware Platforms
- Optimization Techniques
- Benchmarks & Datasets
- Applications & Use Cases
- Community & Resources
- Contributing
🔹 Introduction
The next generation of AI agents is multimodal — capable of understanding and generating text, images, speech, video, and embodied interactions.
Running these models on mobile and edge devices unlocks:
- Privacy: data stays on-device
- Low latency: real-time interaction without cloud roundtrips
- Accessibility: AI everywhere, even offline
- Efficiency: tailored for constrained environments
This repo tracks the latest progress in making multimodal AI efficient, deployable, and agent-ready on edge hardware.
| Device | What fits |
|---|---|
| Phone / NPU | 1–8B INT4 SLM/VLM, ASR, GUI agents |
| Jetson / SBC | 7–32B local agents, VLA |
| MCU / watch | KWS / tiny ASR; LLM on phone or cloud |
📄 Papers
🔖 Surveys & Overviews
| Title | Venue | Year | Materials | Description |
|---|---|---|---|---|
| A Comprehensive Survey on On-Device AI Models | ACM CSUR | 2025 | Paper | Models, systems, and deployment for on-device AI. |
| Mobile Edge Intelligence for Large Language Models | IEEE COMST | 2025 | Paper | LLMs at the mobile edge: latency, offload, serving. |
| Efficient Multimodal Large Language Models: A Survey | arXiv | 2024 | Paper | MLLM compression, vision tokens, and serving. |
| Efficient Diffusion Models: A Survey | arXiv | 2025 | Paper | Algorithm + systems view of fast diffusion. |
| A Survey of Resource-efficient LLM and Multimodal Foundation Models | arXiv | 2024 | Paper | Memory, compute, and energy across the LLM/MLLM stack. |
| Personal LLM Agents: Insights and Survey | arXiv | 2024 | Paper | Capability, efficiency, and security of personal agents. |
🧠 LLM Inference on Edge
| Title | Venue | Year | Materials | Description |
|---|---|---|---|---|
| LLM as a System Service on Mobile Devices | MobiCom | 2024 | Paper | OS-level KV cache, swapping, and multi-app sharing. |
| Large Language Models on Mobile Devices: Measurements & Optimizations | MobiSys | 2024 | Paper | Phone-side latency, energy, and thermal study. |
| PowerInfer-2: Fast Large Language Model Inference on a Smartphone | arXiv | 2024 | Paper | Code | Neuron-cluster sparsity; 47B model on a phone. |
| BitNet b1.58 / BitNet.cpp | arXiv | 2024 | Paper | Code | 1.58-bit ternary LLMs; CPU-first edge inference. |
| Qwen2.5-Omni Technical Report | arXiv | 2025 | Paper | Streaming omni model (text/image/audio/video → text+speech). |
| MobileLLM | CVPR | 2024 | Paper | Code | Sub-billion LLMs designed for phones, not just distilled. |
| EdgeMoE | MobiCom | 2024 | Paper | Expert-wise paging so MoE models fit in phone DRAM. |
| Transformer-Lite | arXiv | 2024 | Paper | High-efficiency LLM decode on phone GPUs (Qualcomm / MTK). |
| LLMCad | arXiv | 2023 | Paper | On-device speculative collaboration; up to 9.3× faster generation. |
🖼️ Multimodal & Generative Models
| Title | Venue | Year | Materials | Description |
|---|---|---|---|---|
| MobileCLIP / MobileCLIP2 | CVPR | 2024–25 | Paper | Code | Fast image–text models; iPhone-class latency. |
| MiniCPM-V / MiniCPM-o | Nat. Commun. | 2025 | Paper | Code | Strong on-device MLLM; GPT-4V-level at phone scale. |
| InternVL / InternVL2-Mobile | CVPR | 2024 | Paper | Code | Open VLM family with mobile-size checkpoints. |
| LLaVA-Mini (1 vision token) | arXiv | 2025 | Paper | Extreme vision-token compression for LMMs. |
| MobileVLM / MobileVLM V2 | arXiv | 2024 | Paper | Code | VLM tuned for mobile throughput. |
| FastVLM | CVPR | 2025 | Paper | Code | Apple hybrid vision encoder; Time-to-First-Token on iPhone. |
| EdgeSAM | ICCV | 2023 | Paper | Proj | Distilled SAM at 30+ FPS on iPhone 14. |
| SnapFusion / MobileDiffusion | ICML / arXiv | 2023–24 | Paper | <2s on-device text-to-image. |
| SDXL-Turbo / LCM | ICLR | 2024 | Paper | 1–4 step diffusion; practical mobile T2I backbone. |
🔊 Speech & Audio
| Title | Venue | Year | Materials | Description |
|---|---|---|---|---|
| Whisper / distil-whisper | ICML | 2023–24 | Paper | Code | Robust ASR; distilled variants fit phones. |
| WhisperKit / whisper.cpp | GitHub | 2024– | whisper.cpp | WhisperKit | On-device ASR on ANE / CPU / GPU. |
| Moonshine | arXiv | 2024 | Paper | Code | Tiny encoder-decoder ASR for edge, not a Whisper clone. |
| Moshi | arXiv | 2024 | Paper | Code | Full-duplex spoken LM; ~200 ms practical latency. |
| Qwen2-Audio / Qwen2.5-Omni | arXiv | 2024–25 | Paper | Open audio-language models used in edge agents. |
| Piper / Kokoro TTS | GitHub | 2023– | Piper | Kokoro | Fast local neural TTS for phones and SBCs. |
🌎 World Models & Embodied AI
| Title | Venue | Year | Materials | Description |
|---|---|---|---|---|
| AndroidWorld | NeurIPS | 2024 | Paper | Code | 116 Android tasks; the default mobile-agent eval. |
| OSWorld | ICLR | 2025 | Paper | Site | Real desktop OS tasks for computer-use agents. |
| OpenVLA | CoRL | 2024 | Paper | Code | Open vision-language-action policy. |
| π0 / π0.5 (Physical Intelligence) | arXiv | 2024–25 | Paper | Generalist robot VLA; distillation path to edge. |
| TinyVLA | arXiv | 2024 | Paper | Compact VLA for onboard robot compute. |
| Mobile ALOHA | arXiv | 2024 | Paper | Site | Low-cost mobile manipulation + imitation. |
🤖 Agent Systems on Edge
| Title | Venue | Year | Materials | Description |
|---|---|---|---|---|
| MobiAgent | arXiv | 2025 | Paper | Customizable mobile agents + acceleration + bench. |
| EcoAgent | arXiv | 2025 | Paper | Cloud planner + on-device execution/observation. |
| Mobile-Agent-v3 / GUI-Owl | arXiv | 2025 | Paper | Strong open GUI agents on AndroidWorld / OSWorld. |
| UI-TARS | arXiv | 2025 | Paper | Code | End-to-end GUI agent; native screenshot-to-action. |
| AppAgent / AppAgent v2 | arXiv | 2024 | Paper | Code | Learn to operate smartphone apps from screenshots. |
| FlashTTS | arXiv | 2025 | Paper | Test-time scaling for agentic LLMs on edge; +2.2× goodput. |
| AutoDroid / AutoDroid-V2 | MobiCom | 2024–25 | Paper | On-device + LLM hybrid Android task automation. |
| SeeClick / ShowUI | CVPR / arXiv | 2024–25 | Paper | Grounded GUI understanding for click agents. |
🧩 On-Device Models
Small models that are actually used as agent backbones on phones, NPUs, and SBCs.
| Model | Size | Modality | Why it matters on edge | Links |
|---|---|---|---|---|
| Qwen2.5 / Qwen3 (Instruct) | 0.5B–14B | Text | Default open SLM; strong tool use at 1.5–7B | Qwen |
| Llama 3.2 | 1B / 3B | Text | Meta’s phone-first SLMs; ExecuTorch path | Llama |
| Gemma 2 / Gemma 3 | 2B–27B | Text / VLM | Apache-friendly; Gemma 3 has vision at small sizes | Gemma |
| Phi-4-mini / Phi-3.5-mini | ~3.8B | Text | Dense quality at phone RAM | Phi |
| SmolLM2 / SmolVLM | 135M–1.7B | Text / VLM | Hugging Face tiny models; browser + phone | SmolLM |
| MobileLLM | 125M–1B | Text | Architecture search for phones, not just distill | Code |
| MiniCPM-V / MiniCPM-o | ~8B (int4 ~5 GB) | V / A / T | Best-in-class on-device MLLM | Code |
| FastVLM | 0.5B–1.5B | VLM | Apple; low TTFT on iPhone | Code |
| InternVL2-1B/2B/4B | 1–4B | VLM | Open mobile VLM checkpoints | Code |
| BitNet b1.58 2B | 2B (1.58-bit) | Text | CPU-native ternary inference | BitNet |
| Moonshine Tiny/Base | 27M / 61M | ASR | Edge speech without Whisper-scale decode | Code |
| Piper / Kokoro | <100M | TTS | Local voice for agents | Piper |
Quantization cheat sheet: Q4_K_M / AWQ-INT4 is the default phone/Jetson tradeoff. Use Q8 only if quality is the bottleneck; 1.58-bit / 2-bit when RAM is the bottleneck.
⚙️ Frameworks & Inference Engines
LLM / VLM runtimes
| Engine | Best on | Notes |
|---|---|---|
| llama.cpp | CPU, Metal, Vulkan, some NPUs | De-facto portable GGUF stack; multimodal via llava/minicpm |
| MLC-LLM | iOS, Android, WebGPU, Metal | TVM compile-once, run everywhere |
| ExecuTorch | iOS, Android, MCU | PyTorch-native on-device; Llama 3.2 official path |
| vLLM / LMDeploy | Jetson, edge GPU | PagedAttention serving for 7–32B local agents |
| SGLang | Edge GPU | Fast structured decode / tool-calling servers |
| Ollama | Laptop, SBC | Easiest local agent backend |
| MediaPipe LLM Inference | Android, Web | Google’s on-device LLM API |
| MNN | Android, iOS | Alibaba lightweight engine + LLM |
| NCNN | Mobile CPU/GPU | Battle-tested CV/ASR engine |
| ONNX Runtime / ORT-GenAI | Cross-platform | DirectML, CoreML, QNN, TensorRT EPs |
| TensorRT / TensorRT-LLM | NVIDIA dGPU, Jetson | Highest Jetson throughput |
| Core ML + mlx | Apple ANE / GPU | Native Apple Silicon path |
| LiteRT (TFLite) | Android NPU, MCU | Google on-device runtime |
| QNN | Snapdragon NPU | Hexagon offload for phones |
| OpenVINO | Intel NPU/GPU | Laptops, industrial PCs |
| llama.rn | React Native | On-device LLM in mobile apps |
| whisper.cpp / WhisperKit | CPU / ANE | Local ASR |
Agent / app layers
- OpenClaw — Self-hosted multi-agent assistant on Jetson-class boxes
- ForestHub edge-agents — Offline agents on Pi / Jetson / industrial gateways (GPIO, UART, MQTT)
- Airgap — React Native on-device RAG agents (Gemma + llama.rn)
- LangGraph / LlamaIndex — Orchestration that can target a local runtime
- Dify — Visual agent builder, often paired with Ollama on the LAN
- Ivy Tendril — Agentic software factory with parallel Git worktrees (dev-time, not on-device)
- WERR - 2026 · Edge/MCU - Zero-memory, sub-2ms machine-native System-1 reflex decision engine using 24-byte fractal coordinate seeds for air-gapped edge agents. (Paper)
🖥️ Hardware Platforms
| Platform | Typical TOPS / RAM | Sweet spot |
|---|---|---|
| Apple A17–M4 (ANE + Metal) | ~35 TOPS ANE / 8–36 GB | Best phone/laptop UX; Core ML + mlx + llama.cpp Metal |
| Snapdragon 8 Gen 3 / 8 Elite | Hexagon NPU | Android agents; QNN + MNN + MediaPipe |
| MediaTek Dimensity 9300/9400 | APU | Android NPUs via NeuroPilot / LiteRT |
| Google Tensor / Pixel | TPU + GPU | AICore, Gemini Nano, on-device ASR |
| NVIDIA Jetson Orin Nano / NX / AGX | 40–275 TOPS / 8–64 GB | Home/factory agent hubs, VLA, local vLLM |
| Qualcomm Dragonwing / RB3 | Industrial NPU | Robotics, cameras, cars |
| Raspberry Pi 5 + Hailo / Coral | 13–26 TOPS add-on | Cheap always-on vision + SLM offload |
| Intel Core Ultra (Meteor/Lunar Lake NPU) | 10–48 TOPS NPU | PC Copilot+ / OpenVINO agents |
| ESP32-S3 / STM32 + NPU MCUs | mW class | KWS, tiny ASR; cloud or phone for the LLM |
| Galaxy Watch / Wear OS | MB-class | Keyword + tiny binary; LLM in the phone/cloud (see ClawWatch) |
Rule of thumb: put perception (ASR, CLIP, GUI encoder) on the NPU; keep decode of 1–8B SLMs on GPU/ANE/CPU; reserve cloud for rare hard planning.
🛠️ Optimization Techniques
| Category | Methods / Papers | Description | Paper | Code |
|---|---|---|---|---|
| Quantization | GPTQ, AWQ, SmoothQuant, OmniQuant, QuaRot, QLoRA, DoRA | W4/W8A8, group-wise or NF4 quantization; activation-aware scaling; outlier rotation; low-bit PEFT; LoRA decomposition for fine-tuning. | GPTQ / AWQ / SmoothQuant / QuaRot / QLoRA / DoRA | GPTQ / AWQ / SmoothQuant / QuaRot / QLoRA |
| KV-cache Quantization | KVQuant, ZipCache, QAQ | 2–3 bit KV compression with <0.1 perplexity drop; enables million-token context windows and memory savings. | KVQuant / ZipCache / QAQ | KVQuant / ZipCache |
| Pruning & Sparsity | SparseGPT, Wanda, Wanda++, Movement pruning, N:M sparsity | Unstructured/structured sparsity up to 60% with minimal accuracy loss; block- and activation-aware pruning for LLMs. | SparseGPT / Wanda / Movement Pruning | SparseGPT / Wanda |
| Efficient Attention | FlashAttention-3, PagedAttention (vLLM), MQA/GQA | Mixed-precision & warp-specialized kernels; KV cache paging; fewer KV heads for faster decode. | FlashAttention-3 / PagedAttention | FlashAttention / vLLM |
| Speculative & Multi-token Decoding | Medusa, EAGLE, EAGLE-3 | Multi-head speculative decoding; feature- and token-level prediction; 2–3.6× speedup. | Medusa / EAGLE | Medusa / EAGLE |
| Multimodal Compression | ToMe, DynamicViT, LLaVA-Mini | Token merging/pruning for ViTs; dynamic vision token selection; extreme compression (1 vision token vs 576). | ToMe / DynamicViT / LLaVA-Mini | ToMe / LLaVA-Mini |
| Efficient Diffusion | Consistency Models, LCM, LCM-LoRA, ADD, SDXL-Turbo, SnapFusion | Few-step or 1-step generation; distillation & adversarial training; mobile-ready pipelines for <2s inference. | Consistency Models / LCM / ADD / SDXL-Turbo / SnapFusion | LCM / SDXL-Turbo / SnapFusion |
| System-level TTS Optimization | FlashTTS | Fast test-time scaling for agentic LLMs on edge; speculative beam extension, dynamic prefix scheduling, memory-aware model allocation. 2.2× higher goodput, 38–68% latency reduction vs. vLLM. | FlashTTS | – |
| On-device split / offload | EdgeMoE, Transformer-Lite, EcoAgent, LLMCad | Keep perception and decode on-device; page experts; use a tiny draft model; send only hard planning to the cloud. | EdgeMoE / Transformer-Lite / LLMCad | PowerInfer |
| 1-bit / ternary | BitNet b1.58, BitNet.cpp | Ternary weights for CPU-first phones and SBCs when INT4 still does not fit RAM. | BitNet | BitNet.cpp |
📊 Benchmarks & Datasets
Measure task success, not only tokens/s. For agents, log latency, energy, RAM, and thermal throttling on the real device.
| Benchmark | Category | What it actually tests | Link |
|---|---|---|---|
| AndroidWorld | Mobile GUI agent | 116 tasks / 20 Android apps; the default phone-agent eval | GitHub |
| OSWorld | Desktop computer-use | Real Ubuntu/Windows apps; multi-step GUI | Site |
| AITW / AndroidControl | Mobile GUI | Large-scale Android action traces | AITW |
| ScreenSpot / ScreenSpot-Pro | GUI grounding | Click the right widget from a screenshot | ScreenSpot |
| MLPerf Tiny | MCU / TinyML | KWS, VWW, image clf, anomaly; latency + energy | MLCommons |
| MLPerf Client / MLPerf Inference | Client / edge LLM | Laptop and edge-server LLM serving | MLCommons |
| Geekbench AI | Device scoring | CPU / GPU / NPU AI score across phones and PCs | Geekbench AI |
| AI Benchmark | Mobile SoC | Classic mobile NPU/GPU CNN/Transformer suite | ai-benchmark.com |
| OpenCompass / MMBench / MMMU | VLM quality | Use together with on-device TTFT and RAM | OpenCompass |
| AIoTBench | Legacy mobile | Older TFLite / Caffe2 / PyTorch Mobile numbers | arXiv |
Suggested on-device report template: model + quant + runtime + SoC → TTFT, tok/s, RAM peak, battery mAh / 1k tokens, thermal after 5 min.
📱 Applications & Use Cases
| Category | Examples | Description | Links |
|---|---|---|---|
| On-device chat | MobileLLM, Gemma 3, Llama 3.2, Phi-4-mini | Sub-3B / INT4 assistants in the phone process | MobileLLM · Gemma |
| GUI / computer-use agents | UI-TARS, AppAgent, Mobile-Agent, AutoDroid | Screenshot → tap/type on Android or desktop | UI-TARS · AppAgent · Mobile-Agent |
| Speech agents | Whisper.cpp, Moonshine, Moshi, Piper | Always-on ASR + local TTS; duplex where RAM allows | whisper.cpp · Moshi · Piper |
| Wearable voice | ClawWatch (NullClaw + Vosk) | Native watch agent: 2.8 MB Zig binary + 68 MB offline STT; ~1 MB RAM | ClawWatch |
| Home / always-on hubs | OpenClaw, ClawBox | Jetson Orin Nano box: multi-agent, browser, Telegram/WhatsApp | OpenClaw · ClawBox |
| Offline support / RAG | Airgap | React Native + llama.rn; on-device Gemma + MiniSearch RAG | Airgap |
| Industrial / IoT | ForestHub edge-agents | Pi / Jetson / gateways; GPIO, UART, MQTT as first-class nodes | edge-agents |
| Robotics / VLA | OpenVLA, TinyVLA, Mobile ALOHA, π0 | Onboard policies; distill giants down to Jetson-class | OpenVLA · TinyVLA · Mobile ALOHA |
| On-device T2I / avatars | SnapFusion, MobileDiffusion, LCM, NanoAvatar | Few-step image gen; Android talking-head demo | LCM · NanoAvatar |
| Privacy / air-gap | llama.cpp + local RAG | Factories, hospitals, airplanes — no token leaves the LAN | llama.cpp |
| Agent identity (experimental) | TWZRD Agent Intel | MCP trust scoring before agent-to-agent payments | intel.twzrd.xyz |
| Reflex decision workspace | answerr | In-browser & edge dual-process AI: sub-1ms reflex routing before slow LLMs | answerr · answerr.me |
Starter stacks (copy-paste)
| Goal | Stack |
|---|---|
| iPhone chat + vision | FastVLM or MiniCPM-V → Core ML / MLC-LLM / mlx |
| Android GUI agent | UI-TARS or Mobile-Agent-v3 + MediaPipe / MNN INT4 |
| Jetson home agent | Qwen2.5-7B-Instruct AWQ + vLLM or llama.cpp CUDA + Piper |
| Pi always-on sensor agent | Hailo/Coral for vision + 1–3B GGUF on CPU + MQTT tools |
| Watch / MCU | Keyword spotting locally; LLM on the paired phone |
🌍 Community & Resources
Lists & docs
- Awesome Edge AI — broader edge-AI list
- Awesome On-Device LLM — on-device LLM papers
- MLC AI · ONNX · ExecuTorch
- Google AI Edge — LiteRT, MediaPipe, on-device Gemini Nano
- Apple Machine Learning Research — FastVLM, MobileCLIP, mlx
Hardware & products
- ClawBox — pre-configured Jetson assistant
- NVIDIA Jetson · Qualcomm AI Hub
Where to watch
- Venues: MobiCom, MobiSys, ASPLOS, MLSys, NeurIPS, CVPR, CoRL
- Hugging Face collections tagged
on-device,gguf,executorch
🤝 Contributing
PRs welcome. Follow the Awesome List Guidelines.
In scope: on-device / near-device multimodal models, runtimes, hardware, agents, and evals with a link (paper, code, or product).
Out of scope: cloud-only agent frameworks with no edge path; closed demos with no numbers.
Please include: one-line description, year, hardware class (phone / NPU / Jetson / MCU), and a working URL.
⭐️ Inspired by the vision of efficient multimodal agents everywhere — from watches and phones to factory gateways and robots.
ai-agents
edge-ai
edge-ai-agents
multimodal-ai
on-device-ai
Contributors
Similar repos
Curated list of papers, frameworks, benchmarks, and applications for multimodal AI agents (LLMs, text-to-image, speech, world models, etc.) on mobile and edge devices.
39
30 commits
updated Sep 23, 2026
README
Awesome Edge AI for Multimodal Agents
A curated list of papers, frameworks, benchmarks, and applications for efficient multimodal agents (LLMs, text-to-image, speech, world models, etc.) on mobile and edge devices.
Focused on inference engines, optimization, and deployment for real-world use.
📑 Contents
- Introduction
- Papers
- On-Device Models
- Frameworks & Inference Engines
- Hardware Platforms
- Optimization Techniques
- Benchmarks & Datasets
- Applications & Use Cases
- Community & Resources
- Contributing
🔹 Introduction
The next generation of AI agents is multimodal — capable of understanding and generating text, images, speech, video, and embodied interactions.
Running these models on mobile and edge devices unlocks:
- Privacy: data stays on-device
- Low latency: real-time interaction without cloud roundtrips
- Accessibility: AI everywhere, even offline
- Efficiency: tailored for constrained environments
This repo tracks the latest progress in making multimodal AI efficient, deployable, and agent-ready on edge hardware.
| Device | What fits |
|---|---|
| Phone / NPU | 1–8B INT4 SLM/VLM, ASR, GUI agents |
| Jetson / SBC | 7–32B local agents, VLA |
| MCU / watch | KWS / tiny ASR; LLM on phone or cloud |
📄 Papers
🔖 Surveys & Overviews
| Title | Venue | Year | Materials | Description |
|---|---|---|---|---|
| A Comprehensive Survey on On-Device AI Models | ACM CSUR | 2025 | Paper | Models, systems, and deployment for on-device AI. |
| Mobile Edge Intelligence for Large Language Models | IEEE COMST | 2025 | Paper | LLMs at the mobile edge: latency, offload, serving. |
| Efficient Multimodal Large Language Models: A Survey | arXiv | 2024 | Paper | MLLM compression, vision tokens, and serving. |
| Efficient Diffusion Models: A Survey | arXiv | 2025 | Paper | Algorithm + systems view of fast diffusion. |
| A Survey of Resource-efficient LLM and Multimodal Foundation Models | arXiv | 2024 | Paper | Memory, compute, and energy across the LLM/MLLM stack. |
| Personal LLM Agents: Insights and Survey | arXiv | 2024 | Paper | Capability, efficiency, and security of personal agents. |
🧠 LLM Inference on Edge
| Title | Venue | Year | Materials | Description |
|---|---|---|---|---|
| LLM as a System Service on Mobile Devices | MobiCom | 2024 | Paper | OS-level KV cache, swapping, and multi-app sharing. |
| Large Language Models on Mobile Devices: Measurements & Optimizations | MobiSys | 2024 | Paper | Phone-side latency, energy, and thermal study. |
| PowerInfer-2: Fast Large Language Model Inference on a Smartphone | arXiv | 2024 | Paper | Code | Neuron-cluster sparsity; 47B model on a phone. |
| BitNet b1.58 / BitNet.cpp | arXiv | 2024 | Paper | Code | 1.58-bit ternary LLMs; CPU-first edge inference. |
| Qwen2.5-Omni Technical Report | arXiv | 2025 | Paper | Streaming omni model (text/image/audio/video → text+speech). |
| MobileLLM | CVPR | 2024 | Paper | Code | Sub-billion LLMs designed for phones, not just distilled. |
| EdgeMoE | MobiCom | 2024 | Paper | Expert-wise paging so MoE models fit in phone DRAM. |
| Transformer-Lite | arXiv | 2024 | Paper | High-efficiency LLM decode on phone GPUs (Qualcomm / MTK). |
| LLMCad | arXiv | 2023 | Paper | On-device speculative collaboration; up to 9.3× faster generation. |
🖼️ Multimodal & Generative Models
| Title | Venue | Year | Materials | Description |
|---|---|---|---|---|
| MobileCLIP / MobileCLIP2 | CVPR | 2024–25 | Paper | Code | Fast image–text models; iPhone-class latency. |
| MiniCPM-V / MiniCPM-o | Nat. Commun. | 2025 | Paper | Code | Strong on-device MLLM; GPT-4V-level at phone scale. |
| InternVL / InternVL2-Mobile | CVPR | 2024 | Paper | Code | Open VLM family with mobile-size checkpoints. |
| LLaVA-Mini (1 vision token) | arXiv | 2025 | Paper | Extreme vision-token compression for LMMs. |
| MobileVLM / MobileVLM V2 | arXiv | 2024 | Paper | Code | VLM tuned for mobile throughput. |
| FastVLM | CVPR | 2025 | Paper | Code | Apple hybrid vision encoder; Time-to-First-Token on iPhone. |
| EdgeSAM | ICCV | 2023 | Paper | Proj | Distilled SAM at 30+ FPS on iPhone 14. |
| SnapFusion / MobileDiffusion | ICML / arXiv | 2023–24 | Paper | <2s on-device text-to-image. |
| SDXL-Turbo / LCM | ICLR | 2024 | Paper | 1–4 step diffusion; practical mobile T2I backbone. |
🔊 Speech & Audio
| Title | Venue | Year | Materials | Description |
|---|---|---|---|---|
| Whisper / distil-whisper | ICML | 2023–24 | Paper | Code | Robust ASR; distilled variants fit phones. |
| WhisperKit / whisper.cpp | GitHub | 2024– | whisper.cpp | WhisperKit | On-device ASR on ANE / CPU / GPU. |
| Moonshine | arXiv | 2024 | Paper | Code | Tiny encoder-decoder ASR for edge, not a Whisper clone. |
| Moshi | arXiv | 2024 | Paper | Code | Full-duplex spoken LM; ~200 ms practical latency. |
| Qwen2-Audio / Qwen2.5-Omni | arXiv | 2024–25 | Paper | Open audio-language models used in edge agents. |
| Piper / Kokoro TTS | GitHub | 2023– | Piper | Kokoro | Fast local neural TTS for phones and SBCs. |
🌎 World Models & Embodied AI
| Title | Venue | Year | Materials | Description |
|---|---|---|---|---|
| AndroidWorld | NeurIPS | 2024 | Paper | Code | 116 Android tasks; the default mobile-agent eval. |
| OSWorld | ICLR | 2025 | Paper | Site | Real desktop OS tasks for computer-use agents. |
| OpenVLA | CoRL | 2024 | Paper | Code | Open vision-language-action policy. |
| π0 / π0.5 (Physical Intelligence) | arXiv | 2024–25 | Paper | Generalist robot VLA; distillation path to edge. |
| TinyVLA | arXiv | 2024 | Paper | Compact VLA for onboard robot compute. |
| Mobile ALOHA | arXiv | 2024 | Paper | Site | Low-cost mobile manipulation + imitation. |
🤖 Agent Systems on Edge
| Title | Venue | Year | Materials | Description |
|---|---|---|---|---|
| MobiAgent | arXiv | 2025 | Paper | Customizable mobile agents + acceleration + bench. |
| EcoAgent | arXiv | 2025 | Paper | Cloud planner + on-device execution/observation. |
| Mobile-Agent-v3 / GUI-Owl | arXiv | 2025 | Paper | Strong open GUI agents on AndroidWorld / OSWorld. |
| UI-TARS | arXiv | 2025 | Paper | Code | End-to-end GUI agent; native screenshot-to-action. |
| AppAgent / AppAgent v2 | arXiv | 2024 | Paper | Code | Learn to operate smartphone apps from screenshots. |
| FlashTTS | arXiv | 2025 | Paper | Test-time scaling for agentic LLMs on edge; +2.2× goodput. |
| AutoDroid / AutoDroid-V2 | MobiCom | 2024–25 | Paper | On-device + LLM hybrid Android task automation. |
| SeeClick / ShowUI | CVPR / arXiv | 2024–25 | Paper | Grounded GUI understanding for click agents. |
🧩 On-Device Models
Small models that are actually used as agent backbones on phones, NPUs, and SBCs.
| Model | Size | Modality | Why it matters on edge | Links |
|---|---|---|---|---|
| Qwen2.5 / Qwen3 (Instruct) | 0.5B–14B | Text | Default open SLM; strong tool use at 1.5–7B | Qwen |
| Llama 3.2 | 1B / 3B | Text | Meta’s phone-first SLMs; ExecuTorch path | Llama |
| Gemma 2 / Gemma 3 | 2B–27B | Text / VLM | Apache-friendly; Gemma 3 has vision at small sizes | Gemma |
| Phi-4-mini / Phi-3.5-mini | ~3.8B | Text | Dense quality at phone RAM | Phi |
| SmolLM2 / SmolVLM | 135M–1.7B | Text / VLM | Hugging Face tiny models; browser + phone | SmolLM |
| MobileLLM | 125M–1B | Text | Architecture search for phones, not just distill | Code |
| MiniCPM-V / MiniCPM-o | ~8B (int4 ~5 GB) | V / A / T | Best-in-class on-device MLLM | Code |
| FastVLM | 0.5B–1.5B | VLM | Apple; low TTFT on iPhone | Code |
| InternVL2-1B/2B/4B | 1–4B | VLM | Open mobile VLM checkpoints | Code |
| BitNet b1.58 2B | 2B (1.58-bit) | Text | CPU-native ternary inference | BitNet |
| Moonshine Tiny/Base | 27M / 61M | ASR | Edge speech without Whisper-scale decode | Code |
| Piper / Kokoro | <100M | TTS | Local voice for agents | Piper |
Quantization cheat sheet: Q4_K_M / AWQ-INT4 is the default phone/Jetson tradeoff. Use Q8 only if quality is the bottleneck; 1.58-bit / 2-bit when RAM is the bottleneck.
⚙️ Frameworks & Inference Engines
LLM / VLM runtimes
| Engine | Best on | Notes |
|---|---|---|
| llama.cpp | CPU, Metal, Vulkan, some NPUs | De-facto portable GGUF stack; multimodal via llava/minicpm |
| MLC-LLM | iOS, Android, WebGPU, Metal | TVM compile-once, run everywhere |
| ExecuTorch | iOS, Android, MCU | PyTorch-native on-device; Llama 3.2 official path |
| vLLM / LMDeploy | Jetson, edge GPU | PagedAttention serving for 7–32B local agents |
| SGLang | Edge GPU | Fast structured decode / tool-calling servers |
| Ollama | Laptop, SBC | Easiest local agent backend |
| MediaPipe LLM Inference | Android, Web | Google’s on-device LLM API |
| MNN | Android, iOS | Alibaba lightweight engine + LLM |
| NCNN | Mobile CPU/GPU | Battle-tested CV/ASR engine |
| ONNX Runtime / ORT-GenAI | Cross-platform | DirectML, CoreML, QNN, TensorRT EPs |
| TensorRT / TensorRT-LLM | NVIDIA dGPU, Jetson | Highest Jetson throughput |
| Core ML + mlx | Apple ANE / GPU | Native Apple Silicon path |
| LiteRT (TFLite) | Android NPU, MCU | Google on-device runtime |
| QNN | Snapdragon NPU | Hexagon offload for phones |
| OpenVINO | Intel NPU/GPU | Laptops, industrial PCs |
| llama.rn | React Native | On-device LLM in mobile apps |
| whisper.cpp / WhisperKit | CPU / ANE | Local ASR |
Agent / app layers
- OpenClaw — Self-hosted multi-agent assistant on Jetson-class boxes
- ForestHub edge-agents — Offline agents on Pi / Jetson / industrial gateways (GPIO, UART, MQTT)
- Airgap — React Native on-device RAG agents (Gemma + llama.rn)
- LangGraph / LlamaIndex — Orchestration that can target a local runtime
- Dify — Visual agent builder, often paired with Ollama on the LAN
- Ivy Tendril — Agentic software factory with parallel Git worktrees (dev-time, not on-device)
- WERR - 2026 · Edge/MCU - Zero-memory, sub-2ms machine-native System-1 reflex decision engine using 24-byte fractal coordinate seeds for air-gapped edge agents. (Paper)
🖥️ Hardware Platforms
| Platform | Typical TOPS / RAM | Sweet spot |
|---|---|---|
| Apple A17–M4 (ANE + Metal) | ~35 TOPS ANE / 8–36 GB | Best phone/laptop UX; Core ML + mlx + llama.cpp Metal |
| Snapdragon 8 Gen 3 / 8 Elite | Hexagon NPU | Android agents; QNN + MNN + MediaPipe |
| MediaTek Dimensity 9300/9400 | APU | Android NPUs via NeuroPilot / LiteRT |
| Google Tensor / Pixel | TPU + GPU | AICore, Gemini Nano, on-device ASR |
| NVIDIA Jetson Orin Nano / NX / AGX | 40–275 TOPS / 8–64 GB | Home/factory agent hubs, VLA, local vLLM |
| Qualcomm Dragonwing / RB3 | Industrial NPU | Robotics, cameras, cars |
| Raspberry Pi 5 + Hailo / Coral | 13–26 TOPS add-on | Cheap always-on vision + SLM offload |
| Intel Core Ultra (Meteor/Lunar Lake NPU) | 10–48 TOPS NPU | PC Copilot+ / OpenVINO agents |
| ESP32-S3 / STM32 + NPU MCUs | mW class | KWS, tiny ASR; cloud or phone for the LLM |
| Galaxy Watch / Wear OS | MB-class | Keyword + tiny binary; LLM in the phone/cloud (see ClawWatch) |
Rule of thumb: put perception (ASR, CLIP, GUI encoder) on the NPU; keep decode of 1–8B SLMs on GPU/ANE/CPU; reserve cloud for rare hard planning.
🛠️ Optimization Techniques
| Category | Methods / Papers | Description | Paper | Code |
|---|---|---|---|---|
| Quantization | GPTQ, AWQ, SmoothQuant, OmniQuant, QuaRot, QLoRA, DoRA | W4/W8A8, group-wise or NF4 quantization; activation-aware scaling; outlier rotation; low-bit PEFT; LoRA decomposition for fine-tuning. | GPTQ / AWQ / SmoothQuant / QuaRot / QLoRA / DoRA | GPTQ / AWQ / SmoothQuant / QuaRot / QLoRA |
| KV-cache Quantization | KVQuant, ZipCache, QAQ | 2–3 bit KV compression with <0.1 perplexity drop; enables million-token context windows and memory savings. | KVQuant / ZipCache / QAQ | KVQuant / ZipCache |
| Pruning & Sparsity | SparseGPT, Wanda, Wanda++, Movement pruning, N:M sparsity | Unstructured/structured sparsity up to 60% with minimal accuracy loss; block- and activation-aware pruning for LLMs. | SparseGPT / Wanda / Movement Pruning | SparseGPT / Wanda |
| Efficient Attention | FlashAttention-3, PagedAttention (vLLM), MQA/GQA | Mixed-precision & warp-specialized kernels; KV cache paging; fewer KV heads for faster decode. | FlashAttention-3 / PagedAttention | FlashAttention / vLLM |
| Speculative & Multi-token Decoding | Medusa, EAGLE, EAGLE-3 | Multi-head speculative decoding; feature- and token-level prediction; 2–3.6× speedup. | Medusa / EAGLE | Medusa / EAGLE |
| Multimodal Compression | ToMe, DynamicViT, LLaVA-Mini | Token merging/pruning for ViTs; dynamic vision token selection; extreme compression (1 vision token vs 576). | ToMe / DynamicViT / LLaVA-Mini | ToMe / LLaVA-Mini |
| Efficient Diffusion | Consistency Models, LCM, LCM-LoRA, ADD, SDXL-Turbo, SnapFusion | Few-step or 1-step generation; distillation & adversarial training; mobile-ready pipelines for <2s inference. | Consistency Models / LCM / ADD / SDXL-Turbo / SnapFusion | LCM / SDXL-Turbo / SnapFusion |
| System-level TTS Optimization | FlashTTS | Fast test-time scaling for agentic LLMs on edge; speculative beam extension, dynamic prefix scheduling, memory-aware model allocation. 2.2× higher goodput, 38–68% latency reduction vs. vLLM. | FlashTTS | – |
| On-device split / offload | EdgeMoE, Transformer-Lite, EcoAgent, LLMCad | Keep perception and decode on-device; page experts; use a tiny draft model; send only hard planning to the cloud. | EdgeMoE / Transformer-Lite / LLMCad | PowerInfer |
| 1-bit / ternary | BitNet b1.58, BitNet.cpp | Ternary weights for CPU-first phones and SBCs when INT4 still does not fit RAM. | BitNet | BitNet.cpp |
📊 Benchmarks & Datasets
Measure task success, not only tokens/s. For agents, log latency, energy, RAM, and thermal throttling on the real device.
| Benchmark | Category | What it actually tests | Link |
|---|---|---|---|
| AndroidWorld | Mobile GUI agent | 116 tasks / 20 Android apps; the default phone-agent eval | GitHub |
| OSWorld | Desktop computer-use | Real Ubuntu/Windows apps; multi-step GUI | Site |
| AITW / AndroidControl | Mobile GUI | Large-scale Android action traces | AITW |
| ScreenSpot / ScreenSpot-Pro | GUI grounding | Click the right widget from a screenshot | ScreenSpot |
| MLPerf Tiny | MCU / TinyML | KWS, VWW, image clf, anomaly; latency + energy | MLCommons |
| MLPerf Client / MLPerf Inference | Client / edge LLM | Laptop and edge-server LLM serving | MLCommons |
| Geekbench AI | Device scoring | CPU / GPU / NPU AI score across phones and PCs | Geekbench AI |
| AI Benchmark | Mobile SoC | Classic mobile NPU/GPU CNN/Transformer suite | ai-benchmark.com |
| OpenCompass / MMBench / MMMU | VLM quality | Use together with on-device TTFT and RAM | OpenCompass |
| AIoTBench | Legacy mobile | Older TFLite / Caffe2 / PyTorch Mobile numbers | arXiv |
Suggested on-device report template: model + quant + runtime + SoC → TTFT, tok/s, RAM peak, battery mAh / 1k tokens, thermal after 5 min.
📱 Applications & Use Cases
| Category | Examples | Description | Links |
|---|---|---|---|
| On-device chat | MobileLLM, Gemma 3, Llama 3.2, Phi-4-mini | Sub-3B / INT4 assistants in the phone process | MobileLLM · Gemma |
| GUI / computer-use agents | UI-TARS, AppAgent, Mobile-Agent, AutoDroid | Screenshot → tap/type on Android or desktop | UI-TARS · AppAgent · Mobile-Agent |
| Speech agents | Whisper.cpp, Moonshine, Moshi, Piper | Always-on ASR + local TTS; duplex where RAM allows | whisper.cpp · Moshi · Piper |
| Wearable voice | ClawWatch (NullClaw + Vosk) | Native watch agent: 2.8 MB Zig binary + 68 MB offline STT; ~1 MB RAM | ClawWatch |
| Home / always-on hubs | OpenClaw, ClawBox | Jetson Orin Nano box: multi-agent, browser, Telegram/WhatsApp | OpenClaw · ClawBox |
| Offline support / RAG | Airgap | React Native + llama.rn; on-device Gemma + MiniSearch RAG | Airgap |
| Industrial / IoT | ForestHub edge-agents | Pi / Jetson / gateways; GPIO, UART, MQTT as first-class nodes | edge-agents |
| Robotics / VLA | OpenVLA, TinyVLA, Mobile ALOHA, π0 | Onboard policies; distill giants down to Jetson-class | OpenVLA · TinyVLA · Mobile ALOHA |
| On-device T2I / avatars | SnapFusion, MobileDiffusion, LCM, NanoAvatar | Few-step image gen; Android talking-head demo | LCM · NanoAvatar |
| Privacy / air-gap | llama.cpp + local RAG | Factories, hospitals, airplanes — no token leaves the LAN | llama.cpp |
| Agent identity (experimental) | TWZRD Agent Intel | MCP trust scoring before agent-to-agent payments | intel.twzrd.xyz |
| Reflex decision workspace | answerr | In-browser & edge dual-process AI: sub-1ms reflex routing before slow LLMs | answerr · answerr.me |
Starter stacks (copy-paste)
| Goal | Stack |
|---|---|
| iPhone chat + vision | FastVLM or MiniCPM-V → Core ML / MLC-LLM / mlx |
| Android GUI agent | UI-TARS or Mobile-Agent-v3 + MediaPipe / MNN INT4 |
| Jetson home agent | Qwen2.5-7B-Instruct AWQ + vLLM or llama.cpp CUDA + Piper |
| Pi always-on sensor agent | Hailo/Coral for vision + 1–3B GGUF on CPU + MQTT tools |
| Watch / MCU | Keyword spotting locally; LLM on the paired phone |
🌍 Community & Resources
Lists & docs
- Awesome Edge AI — broader edge-AI list
- Awesome On-Device LLM — on-device LLM papers
- MLC AI · ONNX · ExecuTorch
- Google AI Edge — LiteRT, MediaPipe, on-device Gemini Nano
- Apple Machine Learning Research — FastVLM, MobileCLIP, mlx
Hardware & products
- ClawBox — pre-configured Jetson assistant
- NVIDIA Jetson · Qualcomm AI Hub
Where to watch
- Venues: MobiCom, MobiSys, ASPLOS, MLSys, NeurIPS, CVPR, CoRL
- Hugging Face collections tagged
on-device,gguf,executorch
🤝 Contributing
PRs welcome. Follow the Awesome List Guidelines.
In scope: on-device / near-device multimodal models, runtimes, hardware, agents, and evals with a link (paper, code, or product).
Out of scope: cloud-only agent frameworks with no edge path; closed demos with no numbers.
Please include: one-line description, year, hardware class (phone / NPU / Jetson / MCU), and a working URL.
⭐️ Inspired by the vision of efficient multimodal agents everywhere — from watches and phones to factory gateways and robots.
ai-agents
edge-ai
edge-ai-agents
multimodal-ai
on-device-ai