Chest X-ray Diagnosis and Treatment Recommendation using AI

Team Members: Pramod Khatri, Sankate Sharma, Ajay Diwakar

Problem Statement:

Current diagnostic systems often analyze chest X-ray images in isolation or utilize text-based electronic health records (EHRs) separately. Combining these modalities can enhance diagnostic accuracy and provide more comprehensive treatment recommendations. Our goal is to develop a multi-modal AI system that integrates Chest X-ray images and EHR text data for diagnosis and treatment recommendation.

Proposed Solution:

The system will:

• Extract features from X-ray images using a vision model (ViT + Decoder).
• Process text inputs using a pre trained LLM (e.g., GPT-2 or ClinicalGPT).
• Combine image and text information using Few-shot learning and fine-tuning of LLaVA-Med.
• Utilize SHAP explainability to provide interpretable outputs.
• Serve results via a FastAPI backend and Streamlit frontend.

Implementation Plan:

Data Preparation:

• Download X-ray images from INSPECT dataset (https://som-shahlab.github.io/inspect-website).
• Pre-process and pair images with text annotations.
• Model Building and Fine-Tuning:
• Fine-tune LLaVA-Med on our dataset using a Few-shot approach.
• Integrate BLIP for image feature extraction and apply SHAP for explainability.

Backend API Development:

• Build a FastAPI endpoint that accepts image and text inputs and returns a diagnosis/treatment.
• Implement a Streamlit-based UI for file uploads, input text, and displaying results.

Testing and Validation:

• Evaluate model performance using validation datasets.

Report and Presentation:

• Prepare a less than 5-page report using the ACM workshop paper template.
• Record a presentation showcasing the project.

Expected Outcome: A multi-modal AI pipeline integrating chest X-ray images and EHR text for enhanced diagnostic accuracy and treatment recommendations with SHAP-based explainability for clinical interpretability.