From Models to Systems: A Comprehensive Survey of Efficient Multimodal Learning

The official GitHub page for the survey paper "From Models to Systems: A Comprehensive Survey of Efficient Multimodal Learning". This paper is accepted by Transactions on Machine Learning Research (TMLR) and awarded the Survey Certificate.

Abstract

The rapid expansion of multimodal models has surfaced formidable bottlenecks in computation, memory, and deployment, catalyzing the rise of Efficient Multimodal Learning (EML) as a pivotal research frontier. Despite intensive progress, a cohesive understanding of what, how, and where efficiency is manifested across the learning stack remains fragmented. This survey systematizes the EML landscape by introducing the first structured, model-to-system taxonomy. We distill insights from over 300 seminal works into three hierarchical levels—model, algorithm, and system—addressing architectural parsimony, execution refinement, and hardware-aware orchestration, respectively. Moving beyond a purely categorical review, we offer a methodological synthesis of the vertical synergies between these layers, elucidating how cross-layer co-design contributes to the fundamental "Efficiency-Utility-Privacy" trade-off. Through an integrative case study of Multimodal Large Language Models (MLLMs), we trace the field’s evolutionary trajectory from initial structural adjustments to modern full-stack resource orchestration. Furthermore, we provide a holistic discussion and application-specific optimization blueprints for diverse domains and posit a paradigm shift toward self-regulating intelligence, where efficiency is an intrinsic, emergent property of the model’s fundamental design rather than a post-hoc constraint. Finally, we present open challenges and future directions that will define the trajectory of EML research. This survey establishes a structured framework for multimodal systems that are not only high-performing and generalizable but natively efficient and ready for ubiquitous deployment. A continuously updated version is available at https://github.com/pwang322/Efficient-Multimodal-Learning-Survey.

Figure 1: Overall landscape of Efficient Multimodal Learning (EML), organized across three interconnected levels—Model, Algorithm (Compression & Acceleration), and System—that jointly optimize architectural design, computation, and deployment, with representative applications illustrated. All these methods will facilitate downstream applications such as Autonomous driving, VR, Robotics, and so on.

Figure 2: The MAS taxonomy for EML with representative works.

Model

Figure 3: Structural paradigms of multimodal encoders. The taxonomy contrasts (left) decoupled modality- specific pipelines utilizing post-hoc alignment or fusion mechanisms with (right) natively unified encoders that collapse heterogeneous signals into a shared parameterized core. This architectural evolution reflects a shift toward functional consolidation, where unification acts as a structural prerequisite for efficiency.

Modality-specific Encoders

Vision Encoder

1. 2017_arXiv_Mobilenets: Efficient convolutional neural networks for mobile vision applications arXiv
2. 2018_CVPR_Shufflenet: An extremely efficient convolutional neural network for mobile devices arXiv
3. 2019_ICML_Efficientnet: Rethinking model scaling for convolutional neural networks arXiv
4. 2021_arXiv_Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer arXiv
5. 2021_NeurIPS_Coatnet: Marrying convolution and attention for all data sizes arXiv
6. 2022_CVPR_Metaformer is actually what you need for vision arXiv
7. 2020_arXiv_An image is worth 16x16 words: Transformers for image recognition at scale arXiv
8. 2021_ICCV_Swin transformer: Hierarchical vision transformer using shifted windows arXivICCV
9. 2021_CVPR_Masked autoencoders are scalable vision learners arXiv
10. 2022_arXiv_Beit v2: Masked image modeling with vector-quantized visual tokenizers arXiv
11. 2023_COLM_Mamba: Linear-Time Sequence Modeling with Selective State Spaces arXiv
12. 2024_arXiv_Vision mamba: Efficient visual representation learning with bidirectional state space model arXiv
13. 2024_arXiv_Kan: Kolmogorov-arnold networks arXiv
14. 2021_ICLR_Learning transferable visual models from natural language supervision arXiv
15. 2021_ICLR_Scaling up visual and vision-language representation learning with noisy text supervision arXiv
16. 2023_ICCV_Sigmoid loss for language image pre-training arXiv
17. 2021_ICCV_Emerging properties in self-supervised vision transformers arXiv
18. 2023_arXiv_Dinov2: Learning robust visual features without supervision arXiv
19. 2025_arXiv_DINOv3 arXiv

Text Encoders

1. 1997_NeuralComputation_Hochreiter, Sepp and Schmidhuber, Jürgen IEEE
2. 2014_EMNLP_Learning phrase representations using RNN encoder-decoder for statistical machine translation arXiv
3. 2018_NAACL_Deep contextualized word representations arXiv
4. 2018_CVPR_Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN arXiv
5. 2016_NeurIPS_LightRNN: Memory and computation-efficient recurrent neural networks arXiv
6. 2017_NeurIPS_Attention is all you need arXiv
7. 2019_NAACL_Bert: Pre-training of deep bidirectional transformers for language understanding arXiv
8. 2020_arXiv_Longformer: The long-document transformer arXiv
9. 2020_NeurIPS_Big Bird: Transformers for Longer Sequences arXiv
10. 2020_arXiv_Reformer: The efficient transformer arXiv
11. 2020_arXiv_Linformer: Self-Attention with Linear Complexity arXiv
12. 2023_arXiv_Llama: Open and efficient foundation language models arXiv
13. 2020_ICLR_Transformers are rnns: Fast autoregressive transformers with linear attention arXiv
14. 2025_arXiv_TextMamba: Scene Text Detector with Mamba arXiv

Audio Encoders

1. 2017_ICASSP_CNN ARCHITECTURES FOR LARGE-SCALE AUDIO CLASSIFICATION arXiv
2. 2020_TASLP_Panns: Large-scale pretrained audio neural networks for audio pattern recognition arXiv
3. 2020_NeurIPS_wav2vec 2.0: a framework for self-supervised learning of speech representations arXiv
4. 2021_TASLP_Hubert: Self-supervised speech representation learning by masked prediction of hidden units arXiv
5. 2021_Interspeech_AST: Audio Spectrogram Transformer arXiv
6. 2022_AAAI_SSAST: Self-Supervised Audio Spectrogram Transformer arXiv
7. 2024_Interspeech_Audio Mamba: Selective State Spaces for Self-Supervised Audio Representations arXix
8. 2022_JSTSP_WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing arXiv

Thermal/Depth/Time-series

1. 2022_CVPR_Target‑aware Dual Adversarial Learning and a Multi‑scenario Multi‑Modality Benchmark to Fuse Infrared and Visible for Object Detection arXiv
2. 2024_CVPR_Flexible Window‑based Self‑attention Transformer in Thermal Image Super‑Resolution CVPR
3. 2023_CVPR_Lite-Mono: A Lightweight CNN and Transformer Architecture for Self-Supervised Monocular Depth Estimation arXiv
4. 2023_ICCV_MonoDETR: Depth‑guided Transformer for Monocular 3D Object Detection arXiv
5. 2021_AAAI_Informer: Beyond efficient transformer for long sequence time-series forecasting arXiv
6. 2023_COLM_Mamba: Linear-Time Sequence Modeling with Selective State Spaces arXiv
7. 2023_ICLR_Liquid Structural State-Space Models arXiv
8. 2023_CVF_Towards zero-shot scale-aware monocular depth estimation arXiv

Unified Encoders

1. 2020_ECCV_Uniter: Universal image-text representation learning arXiv
2. 2021_ICLR_ViLT: Vision‑and‑Language Transformer Without Convolution or Region Supervision arXiv
3. 2023_AAAI_BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learning arXiv
4. 2022_arXiv_Learning audio-visual speech representation by masked multimodal cluster prediction arXiv
5. 2022_CVPR_Flava: A foundational language and vision alignment model arXiv
6. 2022_ICLR_Perceiver IO: A General Architecture for Structured Inputs & Outputs arXiv
7. 2023_WACV_Perceiver-vl: Efficient vision-and-language modeling with iterative latent attention arXiv
8. 2024_NeurIPS_UNIT: Unifying Image and Text Recognition in One Vision Encoder arXiv
9. 2024_ICLR_Emu: Generative Pretraining in Multimodality arXiv
10. 2024_CVPR_Unified‑IO 2: Scaling Autoregressive Multimodal Models with Vision Language Audio and Action arXiv
11. 2024_NAACL_i‑Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data arXiv
12. 2025_arXiv_UGen: Unified Autoregressive Multimodal Model with Progressive Vocabulary Learning arXiv
13. 2024_Grok-1.5 Vision: a preview of xAI’s multimodal model XAI

Structural Sparsity

1. 2022_NeurIPS_Vlmo: Unified vision-language pre-training with mixture-of-modality-experts arXiv
2. 2022_NeurIPS_Multimodal contrastive learning with limoe: the language-image mixture of experts arXiv
3. 2025_TPAMI_Uni-moe: Scaling unified multimodal llms with mixture of experts arXiv
4. 2025_arXiv_LEO-MINI: An Efficient Multimodal Large Language Model using Conditional Token Reduction and Mixture of Multi-Modal Experts arXiv
5. 2024_arXiv_Flex-moe: Modeling arbitrary modality combination via the flexible mixture-of-experts arXiv
6. 2025_CVPR_Nvila: Efficient frontier visual language models arXiv
7. 2025_arXiv_Smolvlm: Redefining small and efficient multimodal models arXiv
8. 2024_arXiv_Llava-mole: Sparse mixture of lora experts for mitigating data conflicts in instruction finetuning mllms arXiv
9. 2024_arXiv_Moe-llava: Mixture of experts for large vision-language models arXiv
10. 2024_NeurIPS_Deer-vla: Dynamic inference of multimodal large language models for efficient robot execution arXiv
11. 2025_arXiv_MoLe-VLA: Dynamic Layer-skipping Vision Language Action Model via Mixture-of-Layers for Efficient Robot Manipulation arXiv
12. 2024_AAAI_EVE: efficient vision-language pre-training with masked prediction and modality-aware moe arXiv
13. 2025_arXiv_Astrea: A MOE-based Visual Understanding Model with Progressive Alignment arXiv
14. 2024_arXiv_Routing experts: Learning to route dynamic experts in multi-modal large language models arXiv
15. 2024_arXiv_Fusemoe: Mixture-of-experts transformers for fleximodal fusion arXiv

Structural Decoding

1. 2022_neurIPS_Flamingo: a visual language model for few-shot learning arXiv
2. 2023_ICLR_Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models arXiv
3. 2023_NeurIPS_Bootstrapping Vision‑Language Learning with Decoupled Language Pre‑training arXiv

Modular Adaptation

1. 2019_ICML_Parameter-efficient transfer learning for NLP arXiv
2. 2022_ICLR_Lora: Low-rank adaptation of large language models arXiv
3. 2022_CVPR_Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks arXiv
4. 2022_NeurIPS_Lst: Ladder side-tuning for parameter and memory efficient transfer learning arXiv
5. 2023_NeurIPS_Visual instruction tuning arXiv
6. 2023_arXiv_Minigpt-4: Enhancing vision-language understanding with advanced large language models arXiv
7. 2025_AAAI_A wander through the multimodal landscape: Efficient transfer learning via low-rank sequence multimodal adapter arXiv
8. 2024_CVPR_Mma: Multi-modal adapter for vision-language models arXiv
9. 2024_arXiv_PaLM2-VAdapter: progressively aligned language model makes a strong vision-language adapter arXiv
10. 2023_ICLR_Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models arXiv
11. 2023_NeurIPS_Cheap and quick: Efficient vision-language instruction tuning for large language models arXiv
12. 2022_ECCV_Tip-adapter: Training-free adaption of clip for few-shot classification arXiv
13. 2023_arXiv_mplug-owl: Modularization empowers large language models with multimodality arXiv
14. 2025_TPAMI_Otter: A multi-modal model with in-context instruction tuning arXiv

Algorithm

Figure 4: Algorithm-level efficiency for refining multimodal execution dynamics. This taxonomy illustrates the modulation of information flow across the EML pipeline through seven primary axes: (i) Token compression and selective computing to filter spatial redundancy and retain informative semantic regions; (ii) Pruning to eliminate structural redundancy within backbone architectures; (iii) Quantization to minimize memory bandwidth via precision discretization; (iv) Knowledge distillation to transfer reasoning behaviors and cognitive patterns to compact learners; (v) Prompting and speculative decoding to streamline input adaptation and parallelize generation; (vi) Caching and reuse to amortize prefill costs through temporal state persistence; and (vii) Runtime sparsity to enable adaptive computation based on input complexity. These strategies transform multimodal execution from static processing to a dynamic, information-flow-aware pipeline.

Token Compression & Selective Computing

1. 2023_arXiv_D2 pruning: Message passing for balancing diversity and difficulty in data pruning arXiv

Tranining-free

1. 2024_arXiv_Sparsevlm: Visual token sparsification for efficient vision-language model inference arXiv
2. 2025_AAAI_ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming arXiv
3. 2025_arXiv_Stop looking for important tokens in multimodal language models: Duplication matters more arXiv
4. 2025_CVPR_Topv: Compatible token pruning with inference time optimization for fast and low-memory multimodal vision language model arXiv
5. 2025_CVPR_Visionzip: Longer is better but not necessary in vision language models arXiv
6. 2025_arXiv_Tokencarve: Information-preserving visual token compression in multimodal large language models arXiv
7. 2024_arXiv_Multi-Stage Vision Token Dropping: Towards Efficient Multimodal Large Language Model arXiv
8. 2025_arXiv_Adaretake: Adaptive redundancy reduction to perceive longer for video-language understanding arXiv
9. 2025_arXiv_What Kind of Visual Tokens Do We Need? Training-free Visual Token Pruning for Multi-modal Large Language Models from the Perspective of Graph arXiv
10. 2024_arXiv_Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction arXiv
11. 2025_AAAI_HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models arXiv
12. 2025_AAAI_Fit and prune: Fast and training-free visual token pruning for multi-modal large language models arXiv
13. 2025_arXiv_Token Sequence Compression for Efficient Multimodal Computing arXiv
14. 2024_NeurIPS_Efficient large multi-modal models via visual context compression arXiv
15. 2023_arXiv_Crossget: Cross-guided ensemble of tokens for accelerating vision-language transformers arXiv
16. 2024_arXiv_Multi-Stage Vision Token Dropping: Towards Efficient Multimodal Large Language Model arXiv
17. 2025_arXiv_What Kind of Visual Tokens Do We Need? Training-free Visual Token Pruning for Multi-modal Large Language Models from the Perspective of Graph arXiv
18. 2025_arXiv_Stop looking for important tokens in multimodal language models: Duplication matters more arXiv
19. 2024_NeurIPS_Efficient large multi-modal models via visual context compression arXiv

Training-based

1. 2023_arXiv_PuMer: Pruning and merging tokens for efficient vision language models arXiv
2. 2024_arXiv_p-MoD: Building Mixture-of-Depths MLLMs via Progressive Ratio Decay arXiv
3. 2025_arXiv_Fast: Efficient action tokenization for vision-language-action models arXiv
4. 2024_arXiv_Tokenpacker: Efficient visual projector for multimodal llm arXiv
5. 2025_arXiv_Folder: Accelerating multi-modal large language models with enhanced performance arXiv
6. 2025_CVPR_Visionzip: Longer is better but not necessary in vision language models arXiv
7. 2024_NeurIPS_Efficient large multi-modal models via visual context compression arXiv
8. 2024_NeurIPS_Deepstack: Deeply stacking visual tokens is surprisingly simple and effective for lmms arXiv
9. 2024_arXiv_FlashSloth: Lightning Multimodal Large Language Models via Embedded Visual Compression arXiv

Pruning

1. 2024_MM_Large multimodal model compression via iterative efficient pruning and distillation arXiv
2. 2023_arXiv_Ecoflap: Efficient coarse-to-fine layer-wise pruning for vision-language models arXiv
3. 2025_arXiv_EfficientLLaVA: Generalizable Auto-Pruning for Large Vision-language Models arXiv
4. 2023_ICLR_UPop: Unified and progressive pruning for compressing vision-language transformers arXiv
5. 2025_arXiv_Modality-Aware Neuron Pruning for Unlearning in Multimodal Large Language Models arXiv
6. 2024_CVPR_MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metric arXiv

Quantization

1. 2024_NeurIPS_Q-vlm: Post-training quantization for large vision-language models arXiv
2. 2025_CVPR_Mbq: Modality-balanced quantization for large vision-language models arXiv
3. 2025_arXiv_Vlmq: Efficient post-training quantization for large vision-language models via hessian augmentation arXiv
4. 2025_ACM_Mquant: Unleashing the inference potential of multimodal large language models via static quaantization arXiv
5. 2025_arXiv_STaMP: Sequence Transformation and Mixed Precision for Low-Precision Activation QuantizationSTaMP: Sequence Transformation and Mixed Precision for Low-Precision Activation Quantization arXiv
6. 2024_MM_Advancing Multimodal Large Language Models with Quantization-Aware Scale Learning for Efficient Adaptation arXiv
7. 2024_NuerIPS_Towards Next-Level Post-Training Quantization of Hyper-Scale Transformers arXiv
8. 2024_MM_Advancing Multimodal Large Language Models with Quantization-Aware Scale Learning for Efficient Adaptation ACM

Knowledge Distillation

Prediction-level

1. 2021_CVPR_Thinking fast and slow: Efficient text-to-visual retrieval with transformers arXiv
2. 2025_arXiv_Simple Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimization arXiv
3. 2024_CVPR_Promptkd: Unsupervised prompt distillation for vision-language models arXiv
4. 2025_arXiv_FDBPL: Faster Distillation-Based Prompt Learning for Region-Aware Vision-Language Models Adaptation arXiv
5. 2025_AAAI_Clip-cid: Efficient clip distillation via cluster-instance discrimination arXiv
6. 2025_ICLR_Open-vocabulary customization from clip via data-free knowledge distillation arXiv
7. 2024_arXiv_LLaVA-KD: A Framework of Distilling Multimodal Large Language Models arXiv
8. 2025_arXiv_MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders arXiv

Representation-level

1. 2024_CVPR_Clip-kd: An empirical study of clip model distillation arXiv
2. 2023_ICCV_Tinyclip: Clip distillation via affinity mimicking and weight inheritance arXiv
3. 2024_arXiv_Dynamic Self-adaptive Multiscale Distillation from Pre-trained Multimodal Large Model for Efficient Cross-modal Representation Learning arXiv
4. 2024_CVPR_Building vision-language models on solid foundations with masked distillation CVPR
5. 2024_CVPR_Mobileclip: Fast image-text models through multi-modal reinforced training arXiv
6. 2025_arXiv_Multimodal mamba: Decoder-only multimodal state space model via quadratic to linear distillation arXiv
7. 2024_arXiv_AMFD: Distillation via Adaptive Multimodal Fusion for Multispectral Pedestrian Detection arXiv
8. 2025_arXiv_Comodo: Cross-modal video-to-imu distillation for efficient egocentric human activity recognition arXiv

Behavior-level

1. 2025_arXiv_Skywork r1v: pioneering multimodal reasoning with chain-of-thought arXiv
2. 2024_CVPR_Visual program distillation: Distilling tools and programmatic reasoning into vision-language models arXiv
3. 2024_arXiv_Llava-mod: Making llava tiny via moe knowledge distillation arXiv

Prompting & Speculative Decoding

1. 2022_ECCV_Visual prompt tuning arXiv
2. 2021_ACL_Prefix-Tuning: Optimizing Continuous Prompts for Generation arXiv
3. 2022_CVPR_Conditional Prompt Learning for Vision-Language Models arXiv
4. 2024_CVPR_On speculative decoding for multimodal large language models arXiv
5. 2025_DAC_AASD: Accelerate Inference by Aligning Speculative Decoding in Multimodal Large Language Models IEEE
6. 2025_arXiv_Speculative decoding and beyond: An in-depth survey of techniques arXiv
7. 2025_arXiv_Spec-vla: speculative decoding for vision-language-action models with relaxed acceptance arXiv

Caching & Reuse

1. 2024_arXiv_VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration arXiv
2. 2024_ECCV_Efficient inference of vision instruction-following models with elastic cache arXiv
3. 2024_arXiv_Look-m: Look-once optimization in kv cache for efficient multimodal long-context inference arXiv
4. 2024_arXiv_Cross-Self KV Cache Pruning for Efficient Vision-Language Inference arXiv
5. 2025_arXiv_Fastcache: Optimizing multimodal llm serving through lightweight kv-cache compression framework arXiv
6. 2025_arXiv_AirCache: Activating Inter-modal Relevancy KV Cache Compression for Efficient Large Vision-Language Model Inference arXiv
7. 2025_arXiv_VLA-Cache: Towards Efficient Vision-Language-Action Model via Adaptive Token Caching in Robotic Manipulation arXiv
8. 2024_arXiv_Inf-MLLM: Efficient streaming inference of multimodal large language models on a single GPU arXiv
9. 2025_arXiv_MPIC: Position-Independent Multimodal Context Caching System for Efficient MLLM Serving arXiv
10. 2025_arXiv_Cache-of-Thought: Master-Apprentice Framework for Cost-Effective Vision Language Model Inference arXiv
11. 2024_arXiv_Prefixkv: Adaptive prefix kv cache is what vision instruction-following models need for efficient generation arXiv
12. 2025_arXiv_Meda: Dynamic kv cache allocation for efficient multimodal long-context inference arXiv

Runtime Sparsity

1. 2025_arXiv_Skip-Vision: Efficient and Scalable Acceleration of Vision-Language Models via Adaptive Token Skipping arXiv
2. 2023_CVPR_Dynamic inference with grounding based vision and language models CVPR
3. 2025_arXiv_MoLe-VLA: Dynamic Layer-skipping Vision Language Action Model via Mixture-of-Layers for Efficient Robot Manipulation arXiv
4. 2024_arXiv_γ-MoD: Exploring Mixture-of-Depth Adaptation for Multimodal Large Language Models arXiv
5. 2020_arXiv_Longformer: The long-document transformer arXiv
6. 2020_NeurIPS_Big Bird: Transformers for Longer Sequences arXiv
7. 2022_arXiv_mplug: Effective and efficient vision-language learning by cross-modal skip-connections arXiv
8. 2022_CVPR_Deecap: Dynamic early exiting for efficient image captioning CVPR
9. 2023_CVPR_You need multiple exiting: Dynamic early exiting for accelerating unified vision language model arXiv
10. 2024_NeurIPS_Deer-vla: Dynamic inference of multimodal large language models for efficient robot execution arXiv
11. 2024_arXiv_Not all layers of llms are necessary during inference arXiv
12. 2024_arXiv_Crema: Generalizable and efficient video-language reasoning via multimodal modular fusion arXiv
13. 2024_arXiv_CAPEEN: Image Captioning with Early Exits and Knowledge Distillation arXiv
14. 2021_NAACL_Multimodal End‑to‑End Sparse Model for Emotion Recognition arXiv
15. 2024_CVPR_Low‑Rank Approximation for Sparse Attention in Multi‑Modal LLMs CVPR
16. 2023_NeurIPS_Ties-merging: Resolving interference when merging models arXiv
17. 2023_arXiv_An empirical study of multimodal model merging arXiv
18. 2024_arXiv_Llava-prumerge: Adaptive token reduction for efficient large multimodal models arXiv
19. 2023_CVPR_You need multiple exiting: Dynamic early exiting for accelerating unified vision language model arXiv
20. 2024_arXiv_Not all layers of llms are necessary during inference arXiv
21. 2025_arXiv_Early Exit and Multi Stage Knowledge Distillation in VLMs for Video Summarization arXiv

System

Figure 5: System-level efficiency for elastic resource orchestration. This framework illustrates the final operationalization of EML, where theoretical gains are translated into realized performance across five primary axes defined in our MAS taxonomy: (i) KV Cache Management and Serving to decouple memory growth from sequence length and optimize throughput; (ii) Edge-cloud Collaboration for establishing hierarchical cognitive pipelines and uncertainty-guided offloading; (iii) Latency-Aware Scheduling and Pipelining to maximize hardware utilization by reordering and overlapping cross-modal requests; and (iv) Hardware-software (HW-SW) Co-design to natively align model architectural topology with the physical constraints of heterogeneous accelerators. Collectively, these strategies transform static multimodal execution into a dynamic, hardware-aware ecosystem.

KV Cache Management & Serving

1. 2025_arXiv_Characterizing and Efficiently Accelerating Multimodal Generation Models arXiv
2. 2025_arXiv_Fastcache: Optimizing multimodal llm serving through lightweight kv-cache compression framework arXiv
3. 2025_arXiv_Meda: Dynamic kv cache allocation for efficient multimodal long-context inference arXiv
4. 2025_arXiv_MPIC: Position-Independent Multimodal Context Caching System for Efficient MLLM Serving arXiv
5. 2024_arXiv_Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context arXiv

Edge–cloud Collaboration

1. 2024_MobiCom_EdgeCloudAI: Edge-Cloud Distributed Video Analytics ACM
2. 2025_arXiv_MoA-Off: Adaptive Heterogeneous Modality-Aware Offloading with Edge-Cloud Collaboration for Efficient Multimodal LLM Inference arXiv
3. 2020_MobiCom_Challenge: COSMOS: A city-scale programmable testbed for experimentation with advanced wireless ACM
4. 2023_CVPR_Cloud-Device Collaborative Adaptation to Continual Changing Environments in the Real-world arXiv

Latency-Aware Scheduling & Pipelining

1. 2025_arXiv_RServe: Overlapping Encoding and Prefill for Efficient LMM Inference arXiv
2. 2024_arXiv_Inf-MLLM: Efficient streaming inference of multimodal large language models on a single GPU arXiv
3. 2023_NeurIPS_H2o: Heavy-hitter oracle for efficient generative inference of large language models arXiv
4. 2025_TAAS_Adaptive Scheduling of Multimodal Large Language Model in Intelligent Edge Computing ACM
5. 2025_arXiv_The Amazon Nova Family of Models: Technical Report and Model Card arXiv
6. 2021_MobiSys_Low-latency Speculative Inference On Distributed Multi-modal Data Streams ACM
7. 2024_NeurIPS_Deer-vla: Dynamic inference of multimodal large language models for efficient robot execution arXiv
8. 2025_CVPR_DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models arXiv
9. 2025_ICML_Streamline Without Sacrifice - Squeeze out Computation Redundancy in LMM arXiv

Hardware-software Co-design

1. 2024_ISPASS_Generative AI Beyond LLMs: System Implications of Multi-Modal Generation arXiv
2. 2023_ACML_Harmonic-NAS: Hardware-Aware Multimodal Neural Architecture Search on Resource-constrained Devices arXiv

Efficient Multimodal Large Language Model

Figure 6: Chronological overview of representative efficient MLLMs. Models are categorized by primary optimization level: model-level (green), algorithm-level (blue), and system-level (orange). This distri- bution highlights a distinct paradigm shift, where model- and algorithm-level optimizations are dominant in the early stages, while system-level resource orchestration has gained significant prominence in recent years.

Application

Affective Computing

1. 2025_AAAI_DLF: Disentangled-language-focused multimodal sentiment analysis arXiv
2. 2025_ICRA_Ugotme: An embodied system for affective human-robot interaction arXiv

Embodied AI & Robotics

1. 2018_NeurIPS_Speaker-follower models for vision-and-language navigation arXiv
2. 2022_CoRL_A persistent spatial semantic representation for high-level natural language instruction execution arXiv
3. 2022_AAAI_Teach: Task-driven embodied agents that chat arXiv
4. 2023_CoRL_Do as i can, not as i say: Grounding language in robotic affordances arXiv
5. 2025_arXiv_CrayonRobo: Object-Centric Prompt-Driven Vision-Language-Action Model for Robotic Manipulation arXiv
6. 2023_CoRL_Rt-2: Vision-language-action models transfer web knowledge to robotic control arXiv
7. 2024_arXiv_Openvla: An open-source vision-language-action model arXiv
8. 2025_arXiv_M3ET: Efficient Vision-Language Learning for Robotics based on Multimodal Mamba-Enhanced Transformer arXiv

Media Understanding and Generation

1. 2021_EMNLP_VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding arXiv
2. 2024_arXiv_Towards Bridging the Cross-modal Semantic Gap for Multi-modal Recommendation arXiv
3. 2023_CVPR_Vop: Text-video co-operative prompt tuning for cross-modal retrieval arXiv
4. 2024_ACL_Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models arXiv
5. 2024_CVPR_MVBench: A Comprehensive Multi-modal Video Understanding Benchmark arXiv
6. 2024_arXiv_Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis arXiv
7. 2022_ECCV_End-to-end active speaker detection arXiv
8. 2023_CVPR_A light weight model for active speaker detection arXiv
9. 2025_arXiv_EMMA: Efficient Multimodal Understanding, Generation, and Editing with a Unified Architecture arXiv

Healthcare

1. 2023_CVPR_Learning to exploit temporal structure for biomedical vision-language processing arXiv
2. 2024_MICCAI_KARGEN: Knowledge-Enhanced Automated Radiology report generation using large language models arXiv
3. 2025_springer_CXR-LLAVA: a multimodal large language model for interpreting chest X-ray images arXiv
4. 2023_NeurIPS_Llava-med: Training a large language-and-vision assistant for biomedicine in one day arXiv
5. 2024_SenSys_Advancing PPG-based continuous blood pressure monitoring from a generative perspective ACM

Spatial Understanding

1. 2019_CVPR_Pointpillars: Fast encoders for object detection from point clouds arXiv
2. 2023_ICRA_BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation arXiv
3. 2021_ITSC_Fusionpainting: Multimodal fusion with adaptive attention for 3d object detection arXiv
4. 2023_ICCV_On-device real-time custom hand gesture recognition arXiv
5. 2025_ICCV_MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs arXiv

Multimodal Reasoning

1. 2019_arXiv_Lxmert: Learning cross-modality encoder representations from transformers arXiv
2. 2020_ECCV_Uniter: Universal image-text representation learning arXiv
3. 2023_IGARSS_Lit-4-rsvqa: Lightweight transformer-based visual question answering in remote sensing arXiv
4. 2023_arXiv_Llava-interactive: An all-in-one demo for image chat, segmentation, generation and editing arXiv
5. 2023_arXiv_Minigpt-4: Enhancing vision-language understanding with advanced large language models arXiv
6. 2024_arXiv_Mathglm-vision: solving mathematical problems with multi-modal large language model arXiv
7. 2023_ICCV_ViperGPT: Visual Inference via Python Execution for Reasoning arXiv
8. 2025_arXiv_Stable and Efficient Single-Rollout RL for Multimodal Reasoning arXiv
9. 2025_arXiv_Towards Efficient Multimodal Unified Reasoning Model via Model Merging arXiv

Federated Learning

1. 2023_ICLR_FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learning arXiv
2. 2023_WWW_pFedPrompt: Learning Personalized Prompt for Vision-Language Models in Federated Learning ACM
3. 2023_ICLR_Multimodal Federated Learning via Contrastive Representation Ensemble arXiv
4. 2025_Neural Networks_FedMEKT: Distillation-Based Embedding Knowledge Transfer for Multimodal Federated Learning arXiv ACM
5. 2021_SIGIR_FedCMR: Federated Cross-Modal Retrieval ACM
6. 2023_arXiv_Multimodal Federated Learning with Missing Modality via Prototype Mask and Contrast arXiv
7. 2024_arXiv_Communication-Efficient Multimodal Federated Learning: Joint Modality and Client Selection arXiv
8. 2022_AAAI_Cross-Modal Federated Human Activity Recognition via Modality-Agnostic and Modality-Specifc Representation Learning AAAI
9. 2024_TPAMI_Cross-Modal Federated Human Activity Recognition IEEE
10. 2020_AAAI_Federated Learning for Vision-and-Language Grounding Problems AAAI
11. 2023_TGRS_FedFusion: Manifold Driven Federated Learning for Multi-satellite and Multi-modality Fusion arXiv

Citation

If you find this repository or survey helpful in your research or work, please cite the following paper:

@article{wang2026models,
  title={From models to systems: A comprehensive survey of efficient multimodal learning},
  author={Wang, Pan and Song, Siwei and Ji, Hui and Cao, Siqi and Yu, Heng and Liu, Zhijian and Yang, Huanrui and Lin, Yingyan Celine and Chen, Beidi and Bansal, Mohit and others},
  journal={Transactions on Machine Learning Research},
  year={2026}
}