Medical Image-to-Text RAG System

This project implements an advanced image-to-text Retrieval-Augmented Generation (RAG) system specifically designed for medical imaging. It utilizes CLIP (Large) for embedding generation and a specialized instruct-tuned Llama3 model (M42-Health/Llama3-Med42-8B) for generating detailed and informative descriptions of medical images. The system integrates datasets such as PathVQA (from Hugging Face) and PAD-UFES-20 (from Kaggle), processing them in a unified pipeline.
The system uses weighted consensus scoring to evaluate and refine candidate text matches. This method not only considers the similarity scores of individual matches but also emphasizes their frequency across multiple candidates. If multiple descriptions align with the image and express the same underlying information, particularly when describing a medical condition or disease shown in the image, this increases confidence in the selected description. This ensures that the generated descriptions are contextually relevant while acknowledging the limitations of AI models, particularly in sensitive domains where the stakes are high, such as healthcare. It represents a reasonable approach to responsibly support professionals in such critical fields.

Project Overview

• Data Preprocessing: Reformats datasets into an image-description format. The PAD-UFES-20 dataset, originally tabular, is converted into paired image-description entries. Similarly, the PathVQA dataset, initially structured as a Q&A database, is transformed into an image-description format for consistent processing.

• Vector Database: Uses KDB.AI to store embeddings of images and text descriptions for efficient similarity-based retrieval.

• Application: Provides a user interface for uploading medical images, retrieving relevant descriptions, and generating detailed medical insights.

System Pipeline

Some results

Here are some examples of descriptions generated by our system :

Some comparisons with LLaVa-1.6-Mistral-7B model :

Data

PAD-UFES-20

The first dataset we used is a dermatological dataset containing skin lesion images with associated tabular metadata. The dataset includes 26 features covering patient medical history, clinical diagnosis, and lesion characteristics. The original metadata (PAD-UFES-20) csv file is transformed from structured format to natural language descriptions this way :

PathVQA

The 2nd dataset is a diverse question answering medical imaging dataset with various types of medical images (organs, tumors, scans, microscopy, medical conditions...) from HuggingFace. Question-answer pairs are converted into descriptive statements to be suitable for our use case :

Folder Structure

medical-image-to-text-rag/
├── data/
│   ├── metadata.csv          # metadata file (PAD-UFES-20 dataset)
│   └── skin_metadata.csv     # Generated after processing
├── src/
│   ├── config.py
│   ├── data_processing.py    # PathVQA dataset
│   ├── dataset_preparation.py
│   ├── vector_database.py
│   └── app.py
├── requirements.txt
└── README.md

Setup Instructions

1. Clone the Repository

git clone https://github.com/MaximeILL/medical-image-to-text-rag.git
cd medical-image-to-text-rag

2. Install Dependencies

Ensure you have Python 3.7 or higher installed.

pip install -r requirements.txt

3. Configure Paths

Review and adjust paths in src/config.py if necessary.

Alternatively, set environment variables to override default paths.

4. Run Data Processing

Process the metadata to generate skin_metadata.csv.

python src/data_processing.py

5. Prepare the Dataset

This step loads the Hugging Face 'PathVQA' dataset and processes it.

python src/dataset_preparation.py

6. Load Embeddings into Vector Database

Ensure you have access to a KDB.AI endpoint and have your API key ready.

Set your KDB.AI credentials:

export KDBAI_ENDPOINT='your_kdbai_endpoint'
export KDBAI_API_KEY='your_kdbai_api_key'

Load embeddings into the vector database:

python src/vector_database.py

Note : KDB.AI API evolves regularly and may be subject to change. Do not hesitate to consult the documentation in case of version or compatibility problems.

7. Run the Application

Launch the user interface to upload images and receive generated descriptions.

python src/app.py

This will start the application. Follow the on-screen instructions to upload an image and receive the generated medical description.

Project Structure Details

src/config.py

Contains configuration variables for the project, such as data paths and KDB AI credentials.

src/data_processing.py

Processes the initial metadata.csv file to generate skin_metadata.csv with enriched descriptions.

src/dataset_preparation.py

• Loads and processes the 'pathvqa' dataset from Hugging Face.
• Turn initial labels into meaningful declarative sentences
• Combines both datasets (PathVQA and PAD-UFES-20) and saves the processed dataset to disk.

src/vector_database.py

• Loads the processed dataset.
• Computes embeddings for images and text descriptions using CLIP.
• Loads embeddings into the KDB.AI vector database for similarity search.

src/app.py

Provides a user interface to:

• Upload an image.
• Process the image to find similar descriptions using the vector database.
• Generate a detailed medical description following our pipeline.

Notes

• Data Download: Images and data are downloaded from the Hugging Face 'pathvqa' dataset automatically during dataset preparation.
• Memory Requirements : the use of models m42-health/Llama3-Med42-8B coupled with CLIP Large require an A100 GPU to run efficiently.

Dependencies

The project requires the following Python packages:

pandas
numpy
torch
Pillow
transformers
datasets
kdbai_client
scikit-learn
tqdm
ipywidgets
ipython
matplotlib

Install them using:

pip install -r requirements.txt

References

• PathVQA dataset: https://huggingface.co/datasets/flaviagiammarino/path-vqa
• PAD-UFES-20 dataset: https://www.kaggle.com/datasets/mahdavi1202/skin-cancer
• KDB AI: https://kdb.ai/

Disclaimer: This project is for educational purposes. The generated medical descriptions should not be used for clinical diagnosis or treatment. Always consult a qualified healthcare professional for medical advice.