Multi-Modal RAG System: big_air_project

A robust Retrieval-Augmented Generation (RAG) system capable of processing and answering complex queries against documents containing text, tables, and images. This project implements a cutting-edge multi-modal indexing and retrieval strategy to enhance the accuracy and grounding of generated responses.

Project Description

This system addresses the limitations of traditional text-only RAG by integrating visual and tabular data into the knowledge base. It is specifically built to handle complex financial or technical documents (e.g., the included qatar_test_doc.pdf) where charts and tables contain critical information that standard language models often miss.

The core innovation lies in the Multi-Modal Indexing strategy, which combines CLIP (for visual semantics) and BGE (for textual context) embeddings into a unified vector space. This ensures high retrieval accuracy across all content modalities (text, tables, and images).

[Image of a diagram showing the fusion of text and image embeddings into a single vector space]

The conversational interface is powered by DSPy, which utilizes a declarative approach to orchestrate the retrieval and generation steps, ensuring faithful citations and providing a three-tier Fallback Mechanism for reliable answers.

Key Features

• Multi-Modal Ingestion: Extracts text, tables (converted to Markdown), and images (captioned via BLIP and OCR via EasyOCR) from PDFs.
• Hybrid Embedding: Uses a weighted fusion ($0.7 \times$ CLIP / $0.3 \times$ BGE) to create a single vector representation for images.
• Declarative AI Logic: Leverages DSPy for structured reasoning steps like Query Analysis and Answer Generation.
• User Interface: A simple web interface built with Flask (app.py) for conversational QA.
• Performance: Demonstrates high retrieval accuracy (up to 89% for text) and low-latency query processing (sub-4s).

---

Technology Stack

---

Project Structure

The key files pushed to the repository are:

---

Installation and Setup

1. Clone the Repository

git clone https://github.com/sujanboseb/big_air_project.git
cd big_air_project

2. Create Virtual Environment

It is highly recommended to use a virtual environment for dependency isolation:

python -m venv venv
# On Windows
.\venv\Scripts\activate
# On macOS/Linux
source venv/bin/activate

3. Install Dependencies

Install all necessary dependencies (assuming they are listed in a requirements.txt file):

pip install -r requirements.txt

4. Configure Environment Variables

Create and populate the .env file at the root of the project with your necessary API keys and configuration parameters.

# .env file content example
GROQ_API_KEY="your_groq_api_key_here"
# ... other model configurations if needed

5. Run the Application

Start the Flask web server:

python app.py

6. Access Credentials

The login interface (Templates/login.html) is configured to accept the following credentials:

Use these credentials to access the RAG Chat interface.

The application will typically start on http://127.0.0.1:5000. Navigate to this address in your browser to begin conversing with the Multi-Modal RAG system.