MSRS: Evaluating Multi-Source Retrieval-Augmented Generation

📄 Paper | 💻 Code

This paper introduces a scalable framework for constructing evaluation benchmarks that challenge RAG systems to integrate information across distinct sources and generate long-form responses. Using our framework, we build two new benchmarks on Multi-Source Retrieval and Synthesis: MSRS-Story and MSRS-Meet.

🚀 Quickstart

Load the corpora for MSRS-Story and MSRS-Meet:

from datasets import load_dataset

story_corpus = load_dataset("yale-nlp/MSRS", "story-corpus", split="corpus")
meeting_corpus = load_dataset("yale-nlp/MSRS", "meeting-corpus", split="corpus")

Corpus Dataset Example:

{
    "id": // Unique ID for the document
    "text": // Document text
}

Load the query-answer pairs for MSRS-Story and MSRS-Meet (available splits: train, test, and validation):

from datasets import load_dataset

story_qa = load_dataset("yale-nlp/MSRS", "story-qa")
meeting_qa = load_dataset("yale-nlp/MSRS", "meeting-qa")

QA Dataset Example:

{
    "id": // Unique ID for the query
    "query": // Query text
    "gold_documents": // List of gold document IDs
    "answer": // List of answer summaries
}