Multimodal RAG PDF Fusion

Evaluation pipeline for comparing methods of combining text and visual data in multimodal Retrieval-Augmented Generation (RAG) systems for PDF documents.

The project compares text-based retrieval (dense, sparse, hybrid), visual retrieval using ColPali-style models (ColQwen2.5, ColSmol-500M), image captioning, combinations of their rankings (RRF, CombMNZ, CombSum), and multimodal reranking (jina-reranker-m0).

Experiments are run on three datasets from the ViDoRe V3 benchmark.

Setup

This project uses uv for dependency management.

1. Install uv following the official instructions.
2. Clone the repository:

   git clone https://github.com/kszabat/multimodal-rag-pdf-fusion.git
   cd multimodal-rag-pdf-fusion

3. Install dependencies:

   uv sync

4. Create a .env file based on .env.example and configure the required environment variables (embedding/VLM model provider, Qdrant connection, etc.).

Notes

• Datasets and vector database collections are not included in the repository due to their size; they are downloaded/generated by running the notebooks in order.
• To use different embedding or vision-language models supported by LlamaIndex, edit utils/config.py and utils/models.py, and install any additional dependencies with uv add <package-name>.