RAG-IGBench: Innovative Evaluation for RAG-based Interleaved Generation in Open-domain Question Answering

🎉 [2025.9.19] RAG-IGBench has been accepted as a poster at NeurIPS 2025 Datasets and Benchmarks Track!

In real-world scenarios, providing user queries with visually enhanced responses can considerably benefit understanding and memory, underscoring the great value of interleaved image-text generation.

Therefore, we present Interleaved Generation based on Retrieval-Augmented Generation (RAG-IG) and the corresponding RAG-IGBench, a thorough benchmark designed specifically to evaluate the task. By integrating MLLMs with the RAG paradigm, we achieve high-quality and semantically coherent image-text interleaved generation.

Specifically, we input the query along with retrieved documents and images into the MLLMs. Through detailed instructions, the MLLMs generate answers in markdown format, incorporating appropriate image indices. Subsequently, we replace these image indices with the corresponding images to produce the final output: a coherent answer where text and images are seamlessly interleaved.

To comprehensively evaluate diverse MLLMs on RAG-IG, we use metrics of three dimensions: text quality, image quality, and image-text consistency. We use ROUGE-1 for text quality evaluation, modified Edit Distance and Kendall Score for image quality evaluation, and CLIP Score and Alignment Score for image-text consistency evaluation. The details of our innovative metrics can be found in our paper.

We provide two versions of the data: RAG_IG_CH and RAG_IG_EN. The Chinese version is sourced from the original social media platform, and AI translates the English version.

The format of each line in RAG_IG_CH.jsonl and RAG_IG_EN.json is:

[
    {
        "id": "a number",
        "query": "user question",
        "documents": [
            "content of doc#1",
            "content of doc#2",
            ...
        ],
        "images":[
            [img_url, img_url, ...],     # images of doc#1
            [img_url, img_url, ...],      # images of doc#2
            ...
        ],
        "gt_raw_answer": "json str",      # model-generated original answer
        "gt_clean_answer": "answer str",      # cleaned markdown-format answer
        "category":[
            "Finance",      # topic of the query
            "what-is"       # one of "what-is", "how-to", "yes-or-no", and "head-to-head"
        ],
        "split": "train/dev"      # data split for training and evaluation
    }
]

For more information, please refer to our 💻Github or 📖Paper.