Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation

This repository is designed to collect and categorize papers related to Multimodal Retrieval-Augmented Generation (RAG) according to our survey paper: Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation. Given the rapid growth in this field, we will continuously update both the paper and this repository to serve as a resource for researchers working on future projects.

📢 News

• January 9, 2026: Happy New Year! We’ve added some new papers in the field.
• September 19, 2025: We've just added new papers to our repository.
• August 20, 2025: The poster and slide for this survey paper have been added to the repository for readers.
• August 1, 2025: We've just added new papers to our repository; a major update!
• June 2, 2025: A new enhanced version of our paper is out now on arXiv! This update also includes new related papers and covers new topics such as agentic interaction and audio-centric retrieval.
• May 15, 2025: This paper has been accepted for publication in the ACL 2025 Findings.
• April 18, 2025: Our website for this topic is up now.
• February 17, 2025: We release the first survey for Multimodal Retrieval-Augmented Generation. Feel free to cite, contribute, or open a pull request to add recent related papers!

📑 List of Contents

• 🔎 General Pipeline
• 🌿 Taxonomy of Recent Advances and Enhancements
• ⚙ Taxonomy of Application Domains
• 📝 Abstract
• 📊 Overview of Popular Datasets
  • 🖼 Image-Text
  • 🎞 Video-Text
  • 🔊 Audio-Text
  • 🩺 Medical
  • 👗 Fashion
  • 💡 QA
  • 🌎 Other
• 📄 Papers
  • 📚 RAG-related Surveys
  • 👓 Retrieval Strategies Advances
    • 🔍 Efficient-Search and Similarity Retrieval
      • ❓ Maximum Inner Product Search-MIPS
      • 💫 Multimodal Encoders
    • 🎨 Modality-Centric Retrieval
      • 📋 Text-Centric
      • 📸 Vision-Centric
      • 🎥 Video-Centric
      • 🎶 Audio-Centric
      • 📰 Document Retrieval and Layout Understanding
    • 🥇🥈 Re-ranking Strategies
      • 🎯 Optimized Example Selection
      • 🧮 Relevance Score Evaluation
      • ⏳ Filtering Mechanisms
  • 🛠 Fusion Mechanisms
    • 🎰 Score Fusion and Alignment
    • ⚔ Attention-Based Mechanisms
    • 🧩 Unified Frameworkes
  • 🚀 Augmentation Techniques
    • 💰 Context-Enrichment
    • 🎡 Adaptive and Iterative Retrieval
    • 🧩 Unified Frameworkes
  • 🤖 Generation Techniques
    • 🧠 In-Context Learning
    • 👨‍⚖️ Reasoning
    • 🤺 Instruction Tuning
    • 📂 Source Attribution and Evidence Transparency
  • 🔧 Training Strategies and Loss Function
  • 🛡️ Robustness and Noise Management
  • 🛠 Tasks Addressed by Multimodal RAGs
  • 📏 Evaluation Metrics
• 🔗 Citations
• 📧 Contact

---

🔎 General Pipeline

🌿 Taxonomy of Recent Advances and Enhancements

⚙ Taxonomy of Application Domains

📝 Abstract

Large Language Models (LLMs) suffer from hallucinations and outdated knowledge due to their reliance on static training data. Retrieval-Augmented Generation (RAG) mitigates these issues by integrating external dynamic information for improved factual grounding. With advances in multimodal learning, Multimodal RAG extends this approach by incorporating multiple modalities such as text, images, audio, and video to enhance the generated outputs. However, cross-modal alignment and reasoning introduce unique challenges beyond those in unimodal RAG.

This survey offers a structured and comprehensive analysis of Multimodal RAG systems, covering datasets, benchmarks, metrics, evaluation, methodologies, and innovations in retrieval, fusion, augmentation, and generation. We precisely review training strategies, robustness enhancements, loss functions, and agent-based approaches, while also exploring the diverse Multimodal RAG scenarios. In addition, we outline open challenges and future directions to guide research in this evolving field. This survey lays the foundation for developing more capable and reliable AI systems that effectively leverage multimodal dynamic external knowledge bases.

📊 Overview of Popular Datasets

🖼 Image-Text

---

🎞 Video-Text

---

🔊 Audio-Text

---

🩺 Medical

---

👗 Fashion

---

💡 QA

---

🌎 Other

---

📄 Papers

📚 RAG-related Surveys

• Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding
• A Comprehensive Survey on Multimodal RAG: All Combinations of Modalities as Input and Output
• RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processing
• Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers
• Retrieval-Augmented Generation for Large Language Models: A Survey
• Retrieval-Augmented Generation for Natural Language Processing: A Survey
• Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG
• Graph Retrieval-Augmented Generation for Large Language Models: A Survey
• Graph Retrieval-Augmented Generation: A Survey
• Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make Your LLMs Use External Data More Wisely
• Trustworthiness in Retrieval-Augmented Generation Systems: A Survey
• A Survey on Retrieval-Augmented Text Generation for Large Language Models
• Searching for Best Practices in Retrieval-Augmented Generation
• Old IR Methods Meet RAG
• A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models
• Benchmarking Large Language Models in Retrieval-Augmented Generation
• A Survey on Retrieval-Augmented Text Generation

👓 Retrieval Strategies Advances

🔍 Efficient-Search and Similarity Retrieval

❓ Maximum Inner Product Search-MIPS

• Fact-Aware Multimodal Retrieval Augmentation for Accurate Medical Radiology Report Generation
• RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval
• ADQ: Adaptive Dataset Quantization
• Efficient and Effective Retrieval of Dense-Sparse Hybrid Vectors using Graph-based Approximate Nearest Neighbor Search
• DeeperImpact: Optimizing Sparse Learned Index Structures
• MUST: An Effective and Scalable Framework for Multimodal Search of Target Modality
• BanditMIPS: Faster Maximum Inner Product Search in High Dimensions
• Revisiting Neural Retrieval on Accelerators
• Query-Aware Quantization for Maximum Inner Product Search
• RA-CM3: Retrieval-Augmented Multimodal Language Modeling
• FARGO: Fast Maximum Inner Product Search via Global Multi-Probing
• MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text
• TPU-KNN: K Nearest Neighbor Search at Peak FLOP/s
• ScaNN: Accelerating large-scale inference with anisotropic vector quantization

💫 Multi-Modal Encoders

• ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering
• ReT-2: Recurrence Meets Transformers for Universal Multimodal Retrieval
• Mi-RAG: Multi-Level Information Retrieval Augmented Generation for Knowledge-based Visual Question Answering
• GME: Improving Universal Multimodal Retrieval by Multimodal LLMs
• VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval
• MARVEL: Unlocking the Multi-Modal Capability of Dense Retrieval via Visual Module Plugin
• Ovis: Structural Embedding Alignment for Multimodal Large Language Model
• UniIR: Training and Benchmarking Universal Multimodal Information Retrievers
• UniVL-DR: Universal Vision-Language Dense Retrieval: Learning A Unified Representation Space for Multi-Modal Retrieval
• InternVideo: General Video Foundation Models via Generative and Discriminative Learning
• FLAVA: A Foundational Language And Vision Alignment Model
• BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
• CLIP: Learning Transferable Visual Models From Natural Language Supervision
• ALIGN: Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

🎨 Modality-Centric Retrieval

📋 Text-Centric

• M²RAG: Multi-modal Retrieval Augmented Multi-modal Generation: Datasets, Evaluation Metrics and Strong Baselines
• OMG-QA: Building Open-Domain Multi-Modal Generative Question Answering Systems
• CRAG: Corrective Retrieval Augmented Generation
• PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers
• XL-HeadTags: Leveraging Multimodal Retrieval Augmentation for the Multilingual Generation of News Headlines and Tags
• RAFT: Adapting Language Model to Domain Specific RAG
• BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
• GTE: Towards General Text Embeddings with Multi-stage Contrastive Learning
• Re-Imagen: Retrieval-Augmented Text-to-Image Generator
• Contriever: Unsupervised Dense Information Retrieval with Contrastive Learning

📸 Vision-Centric

• Enhanced Multimodal RAG-LLM for Accurate Visual Question Answering
• VISA: Retrieval Augmented Generation with Visual Source Attribution
• eCLIP: Improving Medical Multi-modal Contrastive Learning with Expert Annotations
• EchoSight: Advancing Visual-Language Models with Wiki Knowledge
• UniFashion: A Unified Vision-Language Model for Multimodal Fashion Retrieval and Generation
• XL-HeadTags: Leveraging Multimodal Retrieval Augmentation for the Multilingual Generation of News Headlines and Tags
• Visual Delta Generator with Large Multi-modal Models for Semi-supervised Composed Image Retrieval
• VQA4CIR: Boosting Composed Image Retrieval with Visual Question Answering
• RAMM: Retrieval-augmented Biomedical Visual Question Answering with Multi-modal Pre-training
• Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrieval

🎥 Video-Centric

• Q2E: Query-to-Event Decomposition for Zero-Shot Multilingual Text-to-Video Retrieval
• MAGMaR Shared Task System Description: Video Retrieval with OmniEmbed
• VideoRAG: Retrieval-Augmented Generation over Video Corpus
• VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videos
• Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension
• DrVideo: Document Retrieval Based Long Video Understanding
• OmAgent: A Multi-modal Agent Framework for Complex Video Understanding with Task Divide-and-Conquer
• Reversed in Time: A Novel Temporal-Emphasized Benchmark for Cross-Modal Video-Text Retrieval
• iRAG: Advancing RAG for Videos with an Incremental Approach
• InternVideo2: Scaling Foundation Models for Multimodal Video Understanding
• CTCH: Contrastive Transformer Cross-Modal Hashing for Video-Text Retrieval
• Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrieval
• Text Is MASS: Modeling as Stochastic Embedding for Text-Video Retrieval
• MV-Adapter: Multimodal Video Transfer Learning for Video Text Retrieval

🎶 Audio-Centric

• RECAST: Retrieval-Augmented Contextual ASR via Decoder-State Keyword Spotting
• WavRAG: Audio-Integrated Retrieval Augmented Generation for Spoken Dialogue Models
• SpeechRAG: Speech Retrieval-Augmented Generation without Automatic Speech Recognition
• SEAL: Speech Embedding Alignment Learning for Speech Large Language Model with Retrieval-Augmented Generation
• Contextual asr with retrieval augmented large language model.
• RECAP: Retrieval-Augmented Audio Captioning
• Audiobox TTA-RAG: Improving Zero-Shot and Few-Shot Text-To-Audio with Retrieval-Augmented Generation
• DRCap: Decoding CLAP Latents with Retrieval-Augmented Generation for Zero-shot Audio Captioning
• P2PCAP: Enhancing Retrieval-Augmented Audio Captioning with Generation-Assisted Multimodal Querying and Progressive Learning
• LA-RAG:Enhancing LLM-based ASR Accuracy with Retrieval-Augmented Generation
• CA-CLAP: Retrieval Augmented Generation in Prompt-based Text-to-Speech Synthesis with Context-Aware Contrastive Language-Audio Pretraining

📰 Document Retrieval and Layout Understanding

• LILaC: Late Interacting in Layered Component Graph for Open-domain Multimodal Multihop Retrieval
• DREAM: Integrating Hierarchical Multimodal Retrieval with Multi-page Multimodal Language Model for Documents VQA
• One Pic is All it Takes: Poisoning Visual Document Retrieval Augmented Generation with a Single Image
• SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understanding
• ColPali: Efficient Document Retrieval with Vision Language Models
• VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation
• DSE: Unifying Multimodal Retrieval via Document Screenshot Embedding
• M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding
• mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding
• CREAM: Coarse-to-Fine Retrieval and Multi-modal Efficient Tuning for Document VQA
• ColQwen2: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding
• DocLLM: A Layout-Aware Generative Language Model for Multimodal Document Understanding
• Robust Multi Model RAG Pipeline For Documents Containing Text, Table & Images
• ViTLP: Visually Guided Generative Text-Layout Pre-training for Document Intelligence

🥇🥈 Re-ranking Strategies

🎯 Optimized Example Selection

• Hybrid RAG-empowered Multi-modal LLM for Secure Data Management in Internet of Medical Things: A Diffusion-based Contract Approach
• MSIER: How Does the Textual Information Affect the Retrieval of Multimodal In-Context Learning?
• RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Models
• M2-RAAP: A Multi-Modal Recipe for Advancing Adaptation-based Pre-training towards Effective and Efficient Zero-shot Video-text Retrieval
• RAMM: Retrieval-augmented Biomedical Visual Question Answering with Multi-modal Pre-training

🧮 Relevance Score Evaluation

• Re-ranking the Context for Multimodal Retrieval Augmented Generation
• RAG-Check: Evaluating Multimodal Retrieval Augmented Generation Performance
• mR2AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA
• OMG-QA: Building Open-Domain Multi-Modal Generative Question Answering Systems
• RAGTrans: Retrieval-Augmented Hypergraph for Multimodal Social Media Popularity Prediction
• LDRE: LLM-based Divergent Reasoning and Ensemble for Zero-Shot Composed Image Retrieval
• EgoInstructor: Retrieval-Augmented Egocentric Video Captioning
• UniRaG: Unification, Retrieval, and Generation for Multimodal Question Answering With Pre-Trained Language Models

⏳ Filtering Mechanisms

• GME: Improving Universal Multimodal Retrieval by Multimodal LLMs
• MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs
• MuRAR: A Simple and Effective Multimodal Retrieval and Answer Refinement Framework for Multimodal Question Answering
• MAIN-RAG: Multi-Agent Filtering Retrieval-Augmented Generation
• RAFT: Adapting Language Model to Domain Specific RAG

🛠 Fusion Mechanisms

🎰 Score Fusion and Alignment

• UniRAG: Universal Retrieval Augmentation for Multi-Modal Large Language Models
• VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents
• Enhanced Multimodal RAG-LLM for Accurate Visual Question Answering
• MegaPairs: Massive Data Synthesis For Universal Multimodal Retrieval
• Large Language Models Know What is Key Visual Entity: An LLM-assisted Multimodal Retrieval for VQA
• VISA: Retrieval Augmented Generation with Visual Source Attribution
• R2AG: Incorporating Retrieval Information into Retrieval Augmented Generation
• Beyond Text: Optimizing RAG with Multimodal Inputs for Industrial Applications
• RA-BLIP: Multimodal Adaptive Retrieval-Augmented Bootstrapping Language-Image Pre-training
• RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model
• UniRaG: Unification, Retrieval, and Generation for Multimodal Question Answering With Pre-Trained Language Models
• Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs
• MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding
• C3Net: Compound Conditioned ControlNet for Multimodal Content Generation
• REVEAL: Retrieval-Augmented Visual-Language Pre-Training With Multi-Source Multimodal Knowledge Memory
• Re-Imagen: Retrieval-Augmented Text-to-Image Generator

⚔ Attention-Based Mechanisms

• AlzheimerRAG: Multimodal Retrieval Augmented Generation for PubMed articles
• EMERGE: Integrating RAG for Improved Multimodal EHR Predictive Modeling
• Retrieval-Augmented Hypergraph for Multimodal Social Media Popularity Prediction
• M²-RAAP: A Multi-Modal Recipe for Advancing Adaptation-based Pre-training towards Effective and Efficient Zero-shot Video-text Retrieval
• Retrieval-augmented egocentric video captioning
• MORE: Multi-mOdal REtrieval Augmented Generative Commonsense Reasoning
• Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrieval
• MV-Adapter: Multimodal Video Transfer Learning for Video Text Retrieval
• RAMM: Retrieval-augmented Biomedical Visual Question Answering with Multi-modal Pre-training
• MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text

🧩 Unified Frameworks

• M3KG-RAG: Multi-hop Multimodal Knowledge Graph-enhanced Retrieval-Augmented Generation
• LUMA-RAG: Lifelong Multimodal Agents with Provably Stable Streaming Alignment
• ReT-2: Recurrence Meets Transformers for Universal Multimodal Retrieval
• Hybrid RAG-Empowered Multi-Modal LLM for Secure Data Management in Internet of Medical Things: A Diffusion-Based Contract Approach
• M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding
• Self-adaptive Multimodal Retrieval-Augmented Generation
• Iterative Retrieval Augmentation for Multi-Modal Knowledge Integration and Generation
• UFineBench: Towards Text-based Person Retrieval with Ultra-fine Granularity - CVPR 2024
• Simple but Effective Raw-Data Level Multimodal Fusion for Composed Image Retrieval
• PDF-MVQA: A Dataset for Multimodal Information Retrieval in PDF-based Visual Question Answering
• Multimodal Learned Sparse Retrieval with Probabilistic Expansion Control

🚀 Augmentation Techniques

💰 Context-Enrichment

• Enhanced Multimodal RAG-LLM for Accurate Visual Question Answering
• Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension
• Multi-Level Information Retrieval Augmented Generation for Knowledge-based Visual Question Answering
• EMERGE: Enhancing Multimodal Electronic Health Records Predictive Modeling with Retrieval-Augmented Generation
• Img2Loc: Revisiting Image Geolocalization Using Multi-Modality Foundation Models and Image-Based Retrieval-Augmented Generation
• Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs

🎡 Adaptive and Iterative Retrieval

• Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent
• MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models
• mR2AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA
• RAGAR, Your Falsehood Radar: RAG-Augmented Reasoning for Political Fact-Checking using Multimodal Large Language Models
• OMG-QA: Building Open-Domain Multi-Modal Generative Question Answering Systems
• Self-adaptive Multimodal Retrieval-Augmented Generation
• Iterative Retrieval Augmentation for Multi-Modal Knowledge Integration and Generation
• Enhancing Multi-modal Multi-hop Question Answering via Structured Knowledge and Unified Retrieval-Generation

🤖 Generation Techniques

🧠 In-Context Learning

• UniRAG: Universal Retrieval Augmentation for Multi-Modal Large Language Models
• How Does the Textual Information Affect the Retrieval of Multimodal In-Context Learning? (MSIER)
• RAVEN: Multitask Retrieval Augmented Vision-Language Learning
• Retrieval Meets Reasoning: Even High-school Textbook Knowledge Benefits Multimodal Reasoning
• RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model
• Retrieval-Augmented Multimodal Language Modeling (RA-CM3)

👨‍⚖️ Reasoning

• SCMRAG 2.0: Efficient and Scalable Multi-hop Graph RAG with Multimodal Knowledge-Graphs and Agentic Self-Correction
• Progressive Multimodal Reasoning via Active Retrieval
• R2-MultiOmnia: Leading Multilingual Multimodal Reasoning via Self-Training
• MM-Verify: Enhancing Multimodal Reasoning with Chain-of-Thought Verification
• VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation
• RAGAR, Your Falsehood Radar: RAG-Augmented Reasoning for Political Fact-Checking using Multimodal Large Language Models
• Self-adaptive Multimodal Retrieval-Augmented Generation
• LDRE: LLM-based Divergent Reasoning and Ensemble for Zero-Shot Composed Image Retrieval

🤺 Instruction Tuning

• MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale
• MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models
• Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learning
• MegaPairs: Massive Data Synthesis For Universal Multimodal Retrieval
• mR2AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA
• RA-BLIP: Multimodal Adaptive Retrieval-Augmented Bootstrapping Language-Image Pre-training
• Rule: Reliable multimodal rag for factuality in medical vision language models
• MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training (RagVL)
• SURf: Teaching Large Vision-Language Models to Selectively Utilize Retrieved Information
• Visual Delta Generator with Large Multi-modal Models for Semi-supervised Composed Image Retrieval
• InstructBLIP: towards general-purpose vision-language models with instruction tuning

📂 Source Attribution and Evidence Transparency

• MuRAR: A Simple and Effective Multimodal Retrieval and Answer Refinement Framework for Multimodal Question Answering
• VISA: Retrieval Augmented Generation with Visual Source Attribution
• OMG-QA: Building Open-Domain Multi-Modal Generative Question Answering Systems

📂 Agentic Generation and Interaction

• AppAgent v2: Advanced Agent for Flexible Mobile Interactions
• VISA: Retrieval Augmented Generation with Visual Source Attribution
• OMG-QA: Building Open-Domain Multi-Modal Generative Question Answering Systems

🔧 Training Strategies and Loss Function

• FORTIFY: Generative Model Fine-tuning with ORPO for ReTrieval Expansion of InFormal NoisY Text
• Improving Medical Multi-modal Contrastive Learning with Expert Annotations
• EchoSight: Advancing Visual-Language Models with Wiki Knowledge
• UniRaG: Unification, Retrieval, and Generation for Multimodal Question Answering With Pre-Trained Language Models
• HACL: Hallucination Augmented Contrastive Learning for Multimodal Large Language Model
• Multimodal Learned Sparse Retrieval with Probabilistic Expansion Control
• REVEAL: Retrieval-Augmented Visual-Language Pre-Training With Multi-Source Multimodal Knowledge Memory

🛡️ Robustness and Noise Management

• [GenderBias-VL: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing](https://arxiv.org/pdf/2503.13563
• [AlzheimerRAG: Multimodal Retrieval Augmented Generation for PubMed articles]
• MES-RAG: Bringing Multi-modal, Entity-Storage, and Secure Enhancements to RAG
• AlzheimerRAG: Multimodal Retrieval Augmented Generation for PubMed articles
• Quantifying the Gaps Between Translation and Native Perception in Training for Multimodal, Multilingual Retrieval
• RA-BLIP: Multimodal Adaptive Retrieval-Augmented Bootstrapping Language-Image Pre-training
• MORE: Multi-mOdal REtrieval Augmented Generative Commonsense Reasoning
• RAGTrans: Retrieval-Augmented Hypergraph for Multimodal Social Media Popularity Prediction
• MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training (RagVL)
• RA-CM3: Retrieval-Augmented Multimodal Language Modeling
• WFGY Problem Map: Semantic Firewall for RAG Failure Modes

🛠 Tasks Addressed by Multimodal RAGs

• An Intelligent Conversational Agent Using Self-Reflective Retrieval-Augmented Generation for Enhanced Large Language Model Support in National Accounts Learning
• Zero-Shot Anomaly Detection in Laser Powder Bed Fusion Using Multimodal Retrieval-Augmented Generation and Large Language Models
• Specialty-Specific Citation-Enabled AI Clinical Decision Support System for Craniofacial Surgery: Development of CASPE
• Multimodal RAG for Financial Documents: BART-Based Financial Named Entity Recognition and Attention-based Table Parsing for Financial QA Enhancement
• AskHPC: A ChatBot for High Performance Computing User Support
• Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension
• RA-BLIP: Multimodal Adaptive Retrieval-Augmented Bootstrapping Language-Image Pre-training
• Self-adaptive Multimodal Retrieval-Augmented Generation
• Reversed in Time: A Novel Temporal-Emphasized Benchmark for Cross-Modal Video-Text Retrieval
• Mllm is a strong reranker: Advancing multimodal retrieval-augmented generation via knowledge-enhanced reranking and noise-injected training
• Ragar, your falsehood radar: Rag-augmented reasoning for political fact-checking using multimodal large language models
• M2-RAAP: A Multi-Modal Recipe for Advancing Adaptation-based Pre-training towards Effective and Efficient Zero-shot Video-text Retrieval
• RAVEN: Multitask Retrieval Augmented Vision-Language Learning
• UniRaG: Unification, Retrieval, and Generation for Multimodal Question Answering With Pre-Trained Language Models
• A comprehensive survey of hallucination mitigation techniques in large language models
• Ramm: Retrieval-augmented biomedical visual question answering with multi-modal pre-training
• Multi-CLIP: Contrastive Vision-Language Pre-training for Question Answering tasks in 3D Scenes
• Retrieval-augmented multimodal language modeling
• REVEAL: Retrieval-Augmented Visual-Language Pre-Training With Multi-Source Multimodal Knowledge Memory
• Re-imagen: Retrieval-augmented text-to-image generator

🩺 Healthcare and Medicine

• Towards Multimodal Retrieval-Augmented Generation for Medical Visual Question Answering
• MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models
• Fact-aware multimodal retrieval augmentation for accurate medical radiology report generation
• Hybrid RAG-Empowered Multi-Modal LLM for Secure Data Management in Internet of Medical Things: A Diffusion-Based Contract Approach
• RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Models
• AsthmaBot: Multi-modal, Multi-Lingual Retrieval Augmented Generation For Asthma Patient Support
• REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Models

💻 Software Engineering

• Retrieval-Based Prompt Selection for Code-Related Few-Shot Learning
• Docprompting: Generating code by retrieving the docs
• RACE: Retrieval-augmented commit message generation
• Retrieval Augmented Code Generation and Summarization

🕶️ Fashion and E-Commerce

• Unifashion: A unified vision-language model for multimodal fashion retrieval and generation
• Multi-modal Retrieval Augmented Generation for Product Query
• LLM4DESIGN: An Automated Multi-Modal System for Architectural and Environmental Design

🤹 Entertainment and Social Computing

• SoccerRAG: Multimodal Soccer Information Retrieval via Natural Queries
• Predicting Micro-video Popularity via Multi-modal Retrieval Augmentation

🚗 Emerging Applications

• ENWAR: A RAG-empowered Multi-Modal LLM Framework for Wireless Environment Perception
• Beyond Text: Optimizing RAG with Multimodal Inputs for Industrial Applications
• Img2Loc: Revisiting image geolocalization using multi-modality foundation models and image-based retrieval-augmented generation
• RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model

📏 Evaluation Metrics

📊 Retrieval Performance

• Recall@K, Precision@K, F1 Score, and MRR:

  • OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation
  • EMERGE: Integrating RAG for Improved Multimodal EHR Predictive Modeling
  • UniFashion: A Unified Vision-Language Model for Multimodal Fashion Retrieval and Generation
  • Large Language Models Know What is Key Visual Entity: An LLM-assisted Multimodal Retrieval for VQA
  • Multi-Level Information Retrieval Augmented Generation for Knowledge-based Visual Question Answering
  • RAGAR, Your Falsehood Radar: RAG-Augmented Reasoning for Political Fact-Checking using Multimodal Large Language Models
  • Self-adaptive Multimodal Retrieval-Augmented Generation
  • Rule: Reliable multimodal rag for factuality in medical vision language models
  • Iterative Retrieval Augmentation for Multi-Modal Knowledge Integration and Generation
  • MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training (RagVL)
  • M2-RAAP: A Multi-Modal Recipe for Advancing Adaptation-based Pre-training towards Effective and Efficient Zero-shot Video-text Retrieval
  • Simple but Effective Raw-Data Level Multimodal Fusion for Composed Image Retrieval
  • Retrieval-augmented egocentric video captioning
  • UniRaG: Unification, Retrieval, and Generation for Multimodal Question Answering With Pre-Trained Language Models
  • MV-Adapter: Multimodal Video Transfer Learning for Video Text Retrieval
  • Visual Delta Generator with Large Multi-modal Models for Semi-supervised Composed Image Retrieval
  • Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrieval
  • Text Is MASS: Modeling as Stochastic Embedding for Text-Video Retrieval
  • REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Models
  • Multimodal Learned Sparse Retrieval with Probabilistic Expansion Control
  • VQA4CIR: Boosting Composed Image Retrieval with Visual Question Answering
  • MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text

• min(+P, Se):

It represents the minimum value between precision (+P) and sensitivity (Se), providing a balanced measure of model performance.

• EMERGE: Integrating RAG for Improved Multimodal EHR Predictive Modeling
• REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Models

📝 Fluency and Readability

• Fluency (FL):
  • Multi-modal Retrieval Augmented Multi-modal Generation: A Benchmark, Evaluate Metrics and Strong Baselines
  • UniRAG: Universal Retrieval Augmentation for Multi-Modal Large Language Models
  • MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text

✅ Relevance and Accuracy

• Accuracy:
  • Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generation
  • UniRAG: Universal Retrieval Augmentation for Multi-Modal Large Language Models
  • MRAG-BENCH: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models
  • Enhancing Textbook Question Answering Task with Large Language Models and Retrieval Augmented Generation
  • mR2AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA
  • How Does the Textual Information Affect the Retrieval of Multimodal In-Context Learning?
  • RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Models
  • Iterative Retrieval Augmentation for Multi-Modal Knowledge Integration and Generation
  • MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-Augmented Generation via Knowledge-Enhanced Reranking and Noise-Injected Training (RagVL)
  • Advanced Embedding Techniques in Multimodal Retrieval Augmented Generation: A Comprehensive Study on Cross Modal AI Applications
  • RAVEN: Multitask Retrieval Augmented Vision-Language Learning
  • Retrieval Meets Reasoning: Even High-School Textbook Knowledge Benefits Multimodal Reasoning
  • MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text

🖼️ Image-related Metrics

• Fréchet Inception Distance (FID), CLIP Score, Kernel Inception Distance (KID), and Inception Score (IS):

  • UniRAG: Universal Retrieval Augmentation for Multi-Modal Large Language Models
  • UniFashion: A Unified Vision-Language Model for Multimodal Fashion Retrieval and Generation
  • C3Net: Compound Conditioned ControlNet for Multimodal Content Generation
  • Retrieval-Augmented Multimodal Language Modeling (RA-CM3)

• Consensus-Based Image Description Evaluation (CIDEr):

  • UniRAG: Universal Retrieval Augmentation for Multi-Modal Large Language Models
  • RAVEN: Multitask Retrieval Augmented Vision-Language Learning
  • Retrieval-augmented egocentric video captioning
  • RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model
  • Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrieval
  • UniFashion: A Unified Vision-Language Model for Multimodal Fashion Retrieval and Generation
  • MSIER: How Does the Textual Information Affect the Retrieval of Multimodal In-Context Learning?
  • C3Net: Compound Conditioned ControlNet for Multimodal Content Generation
  • Retrieval-Augmented Multimodal Language Modeling (RA-CM3)
  • REVEAL: Retrieval-Augmented Visual-Language Pre-Training With Multi-Source Multimodal Knowledge Memory

• SPICE:

  • UniRAG: Universal Retrieval Augmentation for Multi-Modal Large Language Models

• SPIDEr:

  • C3Net: Compound Conditioned ControlNet for Multimodal Content Generation

🎵 Audio-related Metrics

• Fréchet Audio Distance (FAD), Overall Quality (OVL), and Text Relevenace (REL):
  • C3Net: Compound Conditioned ControlNet for Multimodal Content Generation
  • AudioGen: Textually Guided Audio Generation

🔗 Text Similarity and Overlap Metrics

• BLEU, METEOR, and ROUGE-L:

  • Fact-Aware Multimodal Retrieval Augmentation for Accurate Medical Radiology Report Generation
  • UniRAG: Universal Retrieval Augmentation for Multi-Modal Large Language Models
  • UniFashion: A Unified Vision-Language Model for Multimodal Fashion Retrieval and Generation
  • XL-HeadTags: Leveraging Multimodal Retrieval Augmentation for the Multilingual Generation of News Headlines and Tags
  • Rule: Reliable multimodal rag for factuality in medical vision language models
  • Iterative Retrieval Augmentation for Multi-Modal Knowledge Integration and Generation
  • AsthmaBot: Multi-modal, Multi-Lingual Retrieval Augmented Generation For Asthma Patient Support
  • Advanced Embedding Techniques in Multimodal Retrieval Augmented Generation: A Comprehensive Study on Cross Modal AI Applications
  • RAVEN: Multitask Retrieval Augmented Vision-Language Learning
  • Retrieval-augmented egocentric video captioning
  • RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model
  • Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrieval

• Exact Match (EM):

  • OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation
  • Multi-Level Information Retrieval Augmented Generation for Knowledge-based Visual Question Answering
  • Large Language Models Know What is Key Visual Entity: An LLM-assisted Multimodal Retrieval for VQA
  • Self-adaptive Multimodal Retrieval-Augmented Generation
  • Iterative Retrieval Augmentation for Multi-Modal Knowledge Integration and Generation
  • MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training (RagVL)
  • UniRaG: Unification, Retrieval, and Generation for Multimodal Question Answering With Pre-Trained Language Models
  • MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text

• BERTScore:

  • Fact-Aware Multimodal Retrieval Augmentation for Accurate Medical Radiology Report Generation
  • XL-HeadTags: Leveraging Multimodal Retrieval Augmentation for the Multilingual Generation of News Headlines and Tags

📊 Statistical Metrics

• Spearman’s Rank Correlation (SRC):
  • Predicting Micro-video Popularity via Multi-modal Retrieval Augmentation

⚙️ Efficiency and Computational Performance

• Average Retrieval Time per Query:

  • Advanced Embedding Techniques in Multimodal Retrieval Augmented Generation: A Comprehensive Study on Cross Modal AI Applications

• FLOPs (Floating Point Operations):

  • Multimodal Learned Sparse Retrieval with Probabilistic Expansion Control

• Response Time:

  • Multi-modal Retrieval Augmented Generation for Product Query

• Execution Time:

  • SoccerRAG: Multimodal Soccer Information Retrieval via Natural Queries

• Average Retrieval Number (ARN):

  • Self-adaptive Multimodal Retrieval-Augmented Generation

🏥 Domain-Specific Metrics

• Clinical Relevance (CR):

  • AlzheimerRAG: Multimodal Retrieval Augmented Generation for PubMed articles

• Geodesic Distance:

  • Img2Loc: Revisiting Image Geolocalization Using Multi-Modality Foundation Models and Image-Based Retrieval-Augmented Generation

• GenderBias-VL benchmark:

  • [GenderBias-VL: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing](https://arxiv.org/pdf/2503.13563

---

This README is a work in progress and will be completed soon. Stay tuned for more updates!

---

🔗 Citations

If you find our paper or repository useful, please cite the paper:

@misc{abootorabi2025askmodalitycomprehensivesurvey,
      title={Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation}, 
      author={Mohammad Mahdi Abootorabi and Amirhosein Zobeiri and Mahdi Dehghani and Mohammadali Mohammadkhani and Bardia Mohammadi and Omid Ghahroodi and Mahdieh Soleymani Baghshah and Ehsaneddin Asgari},
      year={2025},
      eprint={2502.08826},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.08826}, 
}

📧 Contact

If you have questions, please send an email to mahdi.abootorabi2@gmail.com.

⭐ Star History