🎉 News

• [2026-05-17] Our MemGraphRAG for memory-enhanced RAG is accepted by KDD'26.
• [2026-04-07] Our ProbeRAG for RAG faithfulness is accepted by ACL'26.
• [2026-04-07] Our BAPO for reliable agentic search is accepted by ACL'26.
• [2026-04-07] Our LegalGraphRAG for reliable legal reasoning is accepted by ACL'26.
• [2026-04-07] Our LogicPoison, a GraphRAG attack model, is accepted by ACL'26.
• [2026-01-26] Our LinearRAG for efficient GraphRAG is accepted by ICLR’26.
• [2026-01-26] Our GraphRAG Benchmark is accepted by ICLR’26.
• [2025-10-27] We release LinearRAG, a relation-free method for efficient GraphRAG.
• [2025-08-24] We support DIGIMON for flexible benchmarking across GraphRAG models.
• [2025-05-25] We release the GraphRAG Benchmark for evaluating GraphRAG models.
• [2025-01-21] We release the GraphRAG survey.

📃 Please cite our paper if you find our survey or repository helpful!

📫 Contact us via emails: {xiangzhishang,wuchuanjie}@stu.xmu.edu.cn, qinggangzhang@jlu.edu.cn

📖 About

This repository is for the GraphRAG-Bench project, a comprehensive benchmark for evaluating Graph Retrieval-Augmented Generation models.

• Introduces Graph Retrieval-Augmented Generation (GraphRAG) concept
• Compares traditional RAG vs GraphRAG approach
• Explains research objective: Identify scenarios where GraphRAG outperforms traditional RAG
• Visual comparison diagram of RAG vs GraphRAG

More Details

Graph retrieval-augmented generation (GraphRAG) has emerged as a powerful paradigm for enhancing large language models (LLMs) with external knowledge. It leverages graphs to model the hierarchical structure between specific concepts, enabling more coherent and effective knowledge retrieval for accurate reasoning. Despite its conceptual promise, recent studies report that GraphRAG frequently underperforms vanilla RAG on many real-world tasks. This raises a critical question: Is GraphRAG really effective, and in which scenarios do graph structures provide measurable benefits for RAG systems? To address this, we propose GraphRAG-Bench, a comprehensive benchmark designed to evaluate GraphRAG models on both hierarchical knowledge retrieval and deep contextual reasoning. GraphRAG-Bench features a comprehensive dataset with tasks of increasing difficulty, covering fact retrieval, complex reasoning, contextual summarization, and creative generation, and a systematic evaluation across the entire pipeline, from graph construction and knowledge retrieval to final generation. Leveraging this novel benchmark, we systematically investigate the conditions when GraphRAG surpasses traditional RAG and the underlying reasons for its success, offering guidelines for its practical application.

🏆 Leaderboards

Two domain-specific leaderboards with comprehensive metrics:

1. GraphRAG-Bench (Novel)

• Evaluates models on literary/fictional content

2. GraphRAG-Bench (Medical)

• Evaluates models on medical/healthcare content

Evaluation Dimensions:

• Fact Retrieval (Accuracy, ROUGE-L)
• Complex Reasoning (Accuracy, ROUGE-L)
• Contextual Summarization (Accuracy, Coverage)
• Creative Generation (Accuracy, Factual Score, Coverage)

🧩 Task Examples

Four difficulty levels with representative examples:

Level 1: Fact Retrieval Example: "Which region of France is Mont St. Michel located?"

Level 2: Complex Reasoning Example: "How did Hinze's agreement with Felicia relate to the perception of England's rulers?"

Level 3: Contextual Summarization Example: "What role does John Curgenven play as a Cornish boatman for visitors exploring this region?"

Level 4: Creative Generation Example: "Retell King Arthur's comparison to John Curgenven as a newspaper article."

🔧 Getting Started

First, install the necessary dependencies for GraphRAG-Bench.

pip install -r requirements.txt

🛠 Installation Guide

To prevent dependency conflicts, we strongly recommend using separate Conda environments for each framework:

We use the installation of LightRAG as an example. For other frameworks, please refer to their respective installation instructions.

# Create and activate environment (example for LightRAG)
conda create -n lightrag python=3.10 -y
conda activate lightrag

# Install LightRAG
git clone https://github.com/HKUDS/LightRAG.git
cd LightRAG
pip install -e .

🚀 Running Examples

We provide detailed instructions on how to use GraphRAG-Bench to evaluate each framework.

Specifically, we introduce how to perform index construction and batch inference for each framework in the Examples folder with instructions in the Examples README.

Note that the evaluation code is standardized across all frameworks to ensure fair comparison. Please refer to the Evaluation folder and the Evaluation README for detailed instructions on the evaluation.

🍀 Citation

If you find this survey helpful, please cite our paper:

@article{xiang2025use,
  title={When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation},
  author={Xiang, Zhishang and Wu, Chuanjie and Zhang, Qinggang and Chen, Shengyuan and Hong, Zijin and Huang, Xiao and Su, Jinsong},
  journal={arXiv preprint arXiv:2506.05690},
  year={2025}
}