Warframe Multi-Modal RAG System

A Multi-Modal Retrieval-Augmented Generation (RAG) pipeline for Warframe knowledge, built for DSAI 413 – Assignment 1. The system indexes curated PDFs (beginner guides, drop tables, fan-kit visuals) using the colSmol-500M ColPali visual-language embedding model, then answers natural-language queries by fusing visual page images, extracted text, and structured table data through Gemini 2.5 Pro.

---

Architecture Overview

fankit/          ──► data_prep.py ──► docs/*.pdf
                                   └──► docs/multimodal_manifest.json
docs/*.pdf       ──► indexer.py   ──► docs/colsmol_index.pt
query            ──► retriever.py ──► Gemini 2.5 Pro ──► answer
                      (hybrid retrieval: ColPali visual + keyword scoring)
app.py           ──► Streamlit UI (wraps retriever)
eval.py          ──► 10-question benchmark suite

Components

---

Setup

Prerequisites

• Python 3.10+
• CUDA-capable GPU recommended (CPU fallback supported)
• A Gemini API Key from Google AI Studio

1. Install dependencies

pip install -r requirements.txt

2. Configure environment

Create a .env file in the project root:

GEMINI_API_KEY=your_api_key_here

3. Prepare documents

Place Warframe PDF documents (e.g. beginner's guide, drop tables) in the docs/ directory.

For fan-kit visuals, ensure the fankit/ directory exists with the official Warframe Fan Kit assets organised by category (Images/, Key Art/, Concept Art/, etc.).

4. Run the data preparation pipeline

python data_prep.py

# Optional: pass a random seed for reproducible fan-kit image selection
python data_prep.py 42

This will:

• Compile fan-kit images into per-category PDFs inside docs/
• Extract text, tables, and image metadata from all PDFs
• Write docs/multimodal_manifest.json

5. Build the index

python indexer.py

Encodes every PDF page with colSmol-500M and saves the index to docs/colsmol_index.pt.
Run data_prep.py first for full multi-modal enrichment; otherwise the indexer falls back to visual-only mode.

6. Launch the Streamlit app

streamlit run app.py

---

Retrieval Strategy

Retrieval uses a hybrid scoring approach:

final_score = 0.75 × visual_score_normalised + 0.25 × keyword_score

Retrieved pages are then passed to Gemini 2.5 Pro with four context layers per page: the visual page image, extracted raw text, structured table data, and embedded image/chart metadata.

---

Evaluation

Run the 10-question benchmark suite (covers table extraction, text analysis, and visual reasoning):

python eval.py

The suite tests:

• Table extraction — Sortie reward pools, rarity tiers, numerical drop chances
• Text analysis — Starter Warframes, movement mechanics, Mastery Rank progression
• Visual reasoning — Fan-kit artwork, faction aesthetics, weapon design details

---

Project Structure

Assignment 1/
├── app.py                        # Streamlit UI
├── data_prep.py                  # Fan-kit compilation + multi-modal manifest extraction
├── indexer.py                    # colSmol-500M page encoding + index builder
├── retriever.py                  # Hybrid retrieval + Gemini generation (WarframeQA)
├── eval.py                       # 10-question benchmark suite
├── requirements.txt
├── .env                          # GEMINI_API_KEY (not committed)
├── docs/
│   ├── *.pdf                     # Source documents (add your own PDFs here)
│   ├── multimodal_manifest.json  # Auto-generated by data_prep.py
│   └── colsmol_index.pt          # Auto-generated by indexer.py
└── fankit/                       # Official Warframe Fan Kit assets
    ├── Images/
    ├── Key Art/
    ├── Concept Art/
    ├── Logos/
    └── ...

---

Requirements

streamlit>=1.42.0
colpali-engine
pymupdf
google-genai>=0.5.0
python-dotenv
pandas
requests
matplotlib
Pillow
lxml
accelerate
tqdm