MiRAGE: A Multiagent Framework for Generating Multimodal Multihop Question-Answer Dataset for RAG Evaluation

MiRAGE is a multi-agent framework for generating high-quality, multimodal, multihop question-answer datasets for evaluating Retrieval-Augmented Generation (RAG) systems.

Interactive Process Flow

Explore the step-by-step multihop QA generation process:

View Interactive Visualization

Multiagent Architecture

Sample QA Pair

Key Features

• Multi-hop Context Completion: Iteratively expands incomplete chunks with relevant context.
• Domain and Expert Role Detection: Automatic domain identification using BERTopic + LLM
• Multi-stage QA Pipeline: Generate, Select, Verify, Correct for quality assurance
• Multimodal Support: Handles text, tables, figures, and images
• Multiple Backend Support: Gemini, OpenAI, and local Ollama models
• Fully Parallelized: Thread and process pools for maximum throughput
• Token Usage Tracking: Automatic tracking of input/output tokens across all LLM calls
• Checkpoint & Resume: Interrupt and resume long-running pipelines without losing progress

Table of Contents

• Installation
• Quick Start
  • Python Library (Recommended)
  • Command Line (CLI)
• Python API Reference
• Examples
• Usage (CLI)
• API Keys Setup
• Configuration
• Command Line Options
• Output Format
• Project Structure
• Contributing
• License

Installation

From PyPI

pip install mirage-benchmark

From Source

git clone https://github.com/ChandanKSahu/MiRAGE.git
cd MiRAGE
pip install -e .

With Optional Dependencies

pip install mirage-benchmark[eval]  # Evaluation metrics (ragas, langchain)
pip install mirage-benchmark[all]   # All optional dependencies

Note: As of v1.2.7, all core dependencies (PDF processing, embeddings, OCR, visualization) are included in the base install. Only evaluation metrics (ragas, langchain) are optional.

GPU Support (FAISS-GPU)

For GPU-accelerated similarity search, install FAISS-GPU via conda:

# Create conda environment (recommended)
conda create -n mirage python=3.11
conda activate mirage

# Install FAISS-GPU
conda install -c pytorch faiss-gpu

# Then install MiRAGE
pip install mirage-benchmark

Quick Start

Step 1: Install the Package

pip install mirage-benchmark

Step 2: Python Library API (Recommended)

Use MiRAGE directly in your Python scripts - just like HuggingFace Transformers or OpenAI:

from mirage import MiRAGE

# Create and run pipeline
pipeline = MiRAGE(
    input_dir="data/my_documents",
    output_dir="output/my_dataset",
    backend="gemini",
    api_key="your-gemini-api-key",
    num_qa_pairs=50,
)
results = pipeline.run()

# Access results
print(f"Generated {len(results)} QA pairs")
for qa in results:
    print(f"Q: {qa['question']}")
    print(f"A: {qa['answer']}\n")

# Save results
results.save("my_dataset.json")

# Convert to pandas DataFrame
df = results.to_dataframe()
df.to_csv("my_dataset.csv")

Advanced Configuration:

from mirage import MiRAGE

# Full control over pipeline
pipeline = MiRAGE(
    input_dir="data/papers",
    output_dir="output/papers_qa",
    backend="gemini",
    api_key="your-key",
    num_qa_pairs=200,
    max_depth=2,
    max_breadth=5,
    embedding_model="nomic",        # "auto", "nomic", "bge_m3", "gemini"
    reranker_model="gemini_vlm",    # "gemini_vlm", "monovlm", "text_embedding"
    device="cuda:0",                # "cuda", "cuda:0", "cpu", or None (auto)
    max_workers=8,
    run_deduplication=True,
    run_evaluation=True,
)

# Or load from config file
pipeline = MiRAGE.from_config("config.yaml", num_qa_pairs=100)

# Method chaining
results = pipeline.configure(num_qa_pairs=50).run()

Load existing results:

from mirage import MiRAGEResults

results = MiRAGEResults.load("output/qa_multihop_pass.json")
print(f"Loaded {len(results)} QA pairs")
df = results.to_dataframe()

Step 3: CLI Usage (Alternative)

You can also use MiRAGE from the command line:

# Set API key
export GEMINI_API_KEY="your-gemini-key"

# Basic usage
run_mirage --input data/my_documents --output output/my_dataset --num-qa-pairs 10

# With API key as argument
run_mirage -i data/my_documents -o output/my_dataset --backend gemini --api-key YOUR_GEMINI_KEY

# Using OpenAI
run_mirage -i data/my_documents -o output/my_dataset --backend openai --api-key YOUR_OPENAI_KEY

# Using local Ollama (no API key needed)
run_mirage -i data/my_documents -o output/my_dataset --backend ollama

# Generate a config file for full customization
run_mirage --init-config

Note: When using --api-key, always specify --backend to indicate which service the key is for.

Step 5: Check Results

ls output/my_dataset/
# qa_multihop_pass.json  - Generated QA pairs (always created)
# chunks.json            - Semantic chunks (always created)
# multihop_visualization.html - Interactive visualization (always created)
# embeddings/            - FAISS index and embeddings

# Optional outputs (if --deduplication and --evaluation flags used):
# qa_deduplicated.json   - Deduplicated QA pairs (with --deduplication)
# evaluation_report.json - Quality metrics (with --evaluation)

Quick Test

# Verify installation
run_mirage --version

# Run preflight checks
run_mirage --preflight

# Generate 1 QA pair for testing
run_mirage --input data/sample --output results/test --num-qa-pairs 1

Usage (CLI)

Basic Usage (QA Generation Only)

By default, MiRAGE runs the core pipeline: document processing, chunking, embedding, and QA generation/verification. Deduplication and evaluation are OFF by default.

# Default: Generates QA pairs without deduplication or evaluation
run_mirage --input <INPUT_DIR> --output <OUTPUT_DIR> --num-qa-pairs 100

With Deduplication

To merge similar QA pairs and remove duplicates:

run_mirage -i data/documents -o output/results --num-qa-pairs 100 --deduplication

With Evaluation Metrics

To compute quality metrics (faithfulness, relevancy, etc.):

run_mirage -i data/documents -o output/results --num-qa-pairs 100 --evaluation

Full Pipeline (Deduplication + Evaluation)

run_mirage -i data/documents -o output/results --num-qa-pairs 100 --deduplication --evaluation

With All Options

run_mirage \
    --input data/documents \
    --output output/results \
    --backend gemini \
    --api-key YOUR_GEMINI_KEY \
    --num-qa-pairs 100 \
    --max-workers 4 \
    --max-depth 2 \
    --embedding-model auto \
    --reranker-model gemini_vlm \
    --deduplication \
    --evaluation \
    --verbose

Auto-Selected Reranker

The reranker is automatically selected based on your backend/API keys:

• Gemini backend/key -> Uses Gemini VLM reranker (fast, API-based, uses same model as VLM config)
• OpenAI backend -> Uses Gemini VLM if Gemini key available, else MonoVLM
• No API keys -> Falls back to MonoVLM (local model, slower)

You can override with --reranker-model flag (options: gemini_vlm, monovlm, text_embedding).

Backend Options:

• gemini (default) - Requires GEMINI_API_KEY or --api-key
• openai - Requires OPENAI_API_KEY or --api-key
• ollama - No API key needed (runs locally)

Pipeline Steps:

Run Preflight Checks

Before running the full pipeline, verify your setup:

run_mirage --preflight

Using Sample Dataset

# Prepare sample data (if you have it)
mkdir -p data/sample
cp /path/to/your/documents/*.pdf data/sample/

# Run on sample
run_mirage -i data/sample -o output/sample_results --num-qa-pairs 10

API Keys Setup

Google Gemini

1. Get API key from: https://makersuite.google.com/app/apikey
2. Set environment variable:

export GEMINI_API_KEY="your-key-here"

Or create a file:

mkdir -p ~/.config/gemini
echo "your-key-here" > ~/.config/gemini/api_key.txt

OpenAI

1. Get API key from: https://platform.openai.com/api-keys
2. Set environment variable:

export OPENAI_API_KEY="your-key-here"

Ollama (Local - Free)

No API key needed! Just install Ollama:

# Install
curl -fsSL https://ollama.com/install.sh | sh

# Start server
ollama serve

# Pull models
ollama pull llama3      # For text
ollama pull llava       # For vision

Configuration

Using config.yaml

Copy the example config and customize:

cp config.yaml.example config.yaml

Edit config.yaml:

backend:
  active: GEMINI  # GEMINI, OPENAI, or OLLAMA
  
  gemini:
    api_key_path: ~/.config/gemini/api_key.txt
    llm_model: gemini-2.0-flash
    vlm_model: gemini-2.0-flash
    
  openai:
    api_key_path: ~/.config/openai/api_key.txt
    llm_model: gpt-4o
    vlm_model: gpt-4o
    
  ollama:
    base_url: http://localhost:11434
    llm_model: llama3
    vlm_model: llava

paths:
  input_pdf_dir: data/documents
  output_dir: output/results

qa_generation:
  target_qa_pairs: 100
  max_workers: 4

Then run:

run_mirage --config config.yaml --input data/documents --output output/results

Note: When installing from pip, you can still use a custom config.yaml file. Place it in your working directory or specify the path with --config.

Cost Optimization

MiRAGE uses LLM/VLM APIs extensively. Two operations consume the most tokens:

1. Document Processing (PDF/HTML -> Markdown -> Chunks)

Cost: High (processes every page with VLM for image/table extraction)

Recommendation:

• Only process documents once on a curated set of relevant files
• Use --skip-pdf-processing and --skip-chunking on subsequent runs
• Pre-filter documents to remove irrelevant content before running MiRAGE

# First run: Process and chunk documents
run_mirage -i data/documents -o output/results --num-qa-pairs 100

# Subsequent runs: Skip processing, only generate QA
run_mirage -i data/documents -o output/results --skip-pdf-processing --skip-chunking --num-qa-pairs 100

2. Multi-hop Context Building

Cost: High (recursive LLM calls to expand context at each depth level)

Recommendation:

• Default: max_depth: 2. Recommended value = 2.
• Higher depth increases time, #tokens, API calls, and consequently costs from API service provider.
• Depth 2 captures most meaningful cross-document relationships with diminishing returns beyond that.

# config.yaml
context:
  max_depth: 2  # Default and recommended. Higher = more time, tokens, API calls, and costs.

Use print_token_stats() or check the pipeline summary to monitor actual token consumption.

Command Line Options

Multihop QA Visualization

Explore an interactive visualization of the multihop QA generation process, showing how context chunks are linked through keywords to generate complex questions:

View Interactive Multihop QA Visualization

The visualization demonstrates:

• Context chunk retrieval and keyword extraction
• Keyword chain relationships across chunks
• Iterative retrieval depth progression
• Final question-answer generation with highlighted concepts

Output Format

Generated Files

output/my_dataset/
├── markdown/              # Converted markdown files
├── chunks.json           # Semantic chunks
├── qa_dataset.json       # Raw QA pairs
├── qa_deduplicated.json  # Final deduplicated QA pairs
├── evaluation_report.json # Quality metrics
└── run_config.json       # Run configuration

QA Dataset Structure

{
  "chunk_id": 1,
  "question": "What is the company's revenue growth?",
  "answer": "The company achieved 15% revenue growth...",
  "context_chunks": [...],
  "hop_count": 2,
  "relevance_score": "9",
  "difficulty_score": "7",
  "expert_persona": "Financial Analyst",
  "domain": "Finance"
}

Multihop QA Visualization

See the Interactive Process Flow at the top of this page for a step-by-step visualization showing:

• Context chunk retrieval and keyword extraction
• Keyword chain relationships across chunks
• Iterative retrieval depth progression
• Final question-answer generation with highlighted concepts

Project Structure

MiRAGE/
├── src/mirage/                    # Main package
│   ├── __init__.py               # Package initialization (exports MiRAGE, MiRAGEConfig, MiRAGEResults)
│   ├── api.py                    # High-level Python API (MiRAGE class)
│   ├── main.py                   # Pipeline orchestration
│   ├── cli.py                    # Command-line interface
│   ├── core/                     # Core functionality
│   │   ├── config.py             # Configuration management
│   │   ├── llm.py                # LLM/VLM API interfaces + token tracking
│   │   └── prompts.py            # Prompt templates
│   ├── embeddings/               # Embedding models
│   │   ├── models.py             # Embedding model selection
│   │   ├── rerankers_multimodal.py  # VLM-based reranking
│   │   └── rerankers_text.py     # Text-based reranking
│   ├── pipeline/                 # Processing pipeline
│   │   ├── pdf_processor.py      # PDF to Markdown conversion
│   │   ├── chunker.py            # Semantic chunking
│   │   ├── context.py            # Multi-hop context retrieval
│   │   ├── qa_generator.py       # QA generation and verification
│   │   ├── domain.py             # Domain/expert extraction
│   │   └── deduplication.py      # QA deduplication
│   ├── evaluation/               # Evaluation metrics
│   │   ├── metrics.py            # Standard RAGAS metrics
│   │   └── metrics_optimized.py  # Optimized metrics (faster)
│   └── utils/                    # Utilities
│       ├── preflight.py          # System checks
│       ├── stats.py              # Dataset statistics
│       ├── ablation.py           # Ablation studies
│       ├── checkpoint.py         # Checkpoint/resume support
│       ├── device.py             # Centralized GPU/CPU device management
│       ├── llm_cache.py          # LLM response caching
│       ├── visualize_multihop.py # Multihop QA visualization
│       └── visualize_pipeline.py # Pipeline flow visualization
├── examples/                      # Ready-to-run sample scripts
│   ├── 01_quick_start.py         # Minimal quick start
│   ├── 02_advanced_config.py     # Full configuration
│   ├── 03_from_config_file.py    # YAML-based config
│   ├── 04_results_analysis.py    # Load, filter, export results
│   ├── 05_openai_backend.py      # Using OpenAI
│   └── 06_method_chaining.py     # Fluent API
├── demo/                          # Interactive demo (Gradio + FastAPI)
├── data/documents/               # Input documents folder
├── output/                       # Generated results
├── assets/                       # Documentation images
├── config.yaml.example           # Example configuration
├── run_mirage.py                 # Main entry point script
├── setup.py                      # Package installation
├── pyproject.toml                # Package configuration
├── requirements.txt              # Dependencies
├── README.md                     # This file
├── CONTRIBUTING.md               # Contribution guidelines
└── LICENSE                       # Apache 2.0 License

Python API Reference

MiRAGE provides a clean Python library API — use it like HuggingFace Transformers or OpenAI. There are two ways to use MiRAGE: the Python library (for custom scripts) and the CLI (for quick terminal usage). Both are fully supported.

Core Classes

MiRAGE Class

from mirage import MiRAGE

# Create a pipeline
pipeline = MiRAGE(
    input_dir="data/my_docs",
    output_dir="output/results",
    backend="gemini",               # "gemini", "openai", "ollama"
    api_key="your-key",
    num_qa_pairs=50,
)

# Run the full pipeline
results = pipeline.run()

# Validate configuration before a long run
pipeline.preflight()  # Returns True/False

# Update settings (supports method chaining)
pipeline.configure(num_qa_pairs=100, max_depth=3)

# Load from a YAML config file
pipeline = MiRAGE.from_config("config.yaml", api_key="your-key")

# Save config for reproducibility
pipeline.save_config("my_experiment.yaml")

MiRAGEResults Class

results = pipeline.run()

# Iterate, index, length
for qa in results:
    print(qa['question'], qa['answer'])

first_qa = results[0]
print(f"Total: {len(results)}")

# Quick access to all questions/answers
print(results.questions)    # ["What is...", "How does...", ...]
print(results.answers)      # ["The answer is...", ...]

# Filter by field
multihop = results.filter(question_type="multihop")
hard = results.filter(difficulty="hard")

# Random sample
subset = results.sample(n=10, seed=42)

# Save (JSON or JSONL)
results.save("dataset.json")
results.save("dataset.jsonl", format="jsonl")

# Load from file
loaded = MiRAGEResults.load("dataset.json")

# Convert to pandas DataFrame
df = results.to_dataframe()
df.to_csv("dataset.csv")

MiRAGEConfig Class

from mirage import MiRAGEConfig

# Create from keyword arguments
config = MiRAGEConfig(backend="gemini", num_qa_pairs=100, device="cuda:0")

# Create from dictionary
config = MiRAGEConfig.from_dict({"backend": "gemini", "num_qa_pairs": 100})

# Load from YAML
config = MiRAGEConfig.from_yaml("config.yaml")

# Save to YAML
config.save_yaml("my_config.yaml")

# Convert to dict
d = config.to_dict()

Configuration Parameters

Low-Level API (Advanced)

For fine-grained control, you can import individual components:

from mirage import run_pipeline, run_preflight_checks
from mirage import get_device, is_gpu_available
from mirage.core.llm import call_llm_simple, get_token_stats, print_token_stats
from mirage.embeddings.models import NomicVLEmbed, get_best_embedding_model
from mirage.pipeline.context import build_complete_context
from mirage.pipeline.domain import fetch_domain_and_role

Examples

MiRAGE comes with ready-to-run example scripts in the examples/ directory:

Example: Quick Start (Python)

from mirage import MiRAGE

# 3 lines to generate a QA dataset
pipeline = MiRAGE(input_dir="data/docs", output_dir="output", backend="gemini", api_key="YOUR_KEY")
results = pipeline.run()
results.save("my_dataset.json")

Example: Quick Start (CLI)

# Same thing from the terminal
export GEMINI_API_KEY="your-key"
run_mirage --input data/docs --output output --num-qa-pairs 10

Example: Advanced (Python)

from mirage import MiRAGE

pipeline = MiRAGE(
    input_dir="data/papers",
    output_dir="output/papers_qa",
    backend="gemini",
    api_key="your-key",
    num_qa_pairs=200,
    max_depth=3,
    embedding_model="nomic",
    device="cuda:0",
    run_deduplication=True,
)
results = pipeline.run()

# Filter and export
multihop = results.filter(question_type="multihop")
df = multihop.to_dataframe()
df.to_csv("multihop_qa.csv")

Example: CLI with All Options

run_mirage \
  --input data/documents \
  --output output/results \
  --backend gemini \
  --api-key YOUR_KEY \
  --num-qa-pairs 100 \
  --max-depth 3 \
  --embedding-model nomic \
  --reranker-model gemini_vlm \
  --max-workers 8 \
  --deduplication \
  --evaluation \
  --verbose

Troubleshooting

Command Not Found

If run_mirage command is not found after pip installation:

# Check if package is installed
pip show mirage-benchmark

# Reinstall if needed
pip install --upgrade mirage-benchmark

# Verify installation
run_mirage --version

API Key Issues

# Check if API key is set
echo $GEMINI_API_KEY  # or $OPENAI_API_KEY

# Set it if missing
export GEMINI_API_KEY="your-key"

Preflight Check Failures

# Run verbose preflight
run_mirage --preflight --verbose

Import Errors (Development)

If you're developing from source and encounter import errors:

# Reinstall in editable mode
pip install -e .

# Or run directly with PYTHONPATH
PYTHONPATH=src python src/mirage/run_mirage.py --help

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Submit a pull request

See CONTRIBUTING.md for details.

License

Apache License 2.0 - see LICENSE