MedIQ

A Production-Ready, 241-Class Multimodal Clinical Intelligence Engine with Real-Time AI Telehealth

MedIQ (formerly DermLIP Server) is a highly optimized, fully localized inference and training engine. It leverages a streamlined architecture (SigLIP2, EVA-CLIP, and OpenAI-CLIP) to classify 241 long-tail dermatology conditions and is scalable to audio and text diagnostic inputs. It completely eliminates Language Model hallucination through a hard-coded Clinical Knowledge Base and a deterministic rule engine.

NEW: Voice Calling Microservice - Real-time WebRTC telehealth consultations with AI-powered transcription, clinical reasoning, and bidirectional voice (patients can speak to AND listen to the AI doctor). 100% local execution with zero external API calls.

Everything is containerized via Docker for safe, isolated execution and connects through a unified FastAPI Gateway.

---

🗺️ Project Navigation & Master Documentation

We maintain an exhaustive suite of documentation for every script, algorithm, and hyperparameter in this project. If you have a question, it is answered in the docs/ folder.

• 📕 Master Project Glossary - Explains every acronym, ML term, and training technique (including AMP, EMA, tail classes).
• 📈 Fine-Tuning Guide (v3) - Massive deep dive into our training philosophy, covering every single hyperparameter in finetune.py.
• 📊 Metrics & Evals Guide - Details on AUROC, Kappa, F1 testing, and the Dashboard structure.
• 🚀 Training Pipeline - One-page architectural map of the dataset-to-checkpoint journey.
• 🖼️ Dataset update pipeline - Scrape → review → merge tail-class images; provenance, dedup, audits, and honest training metrics.
• 🧠 App / Core Engine - Deep dive into the deterministic caching and hierarchical locking algorithm.
• 🛡️ OOD Gate System - The 3-stage Skin & Condition gate system to block non-clinical images.
• 💾 Persistence Layer (Supabase) - Deep dive into the HMS architecture, D2C guest flow, GiST exclusion constraints, and RLS tenant isolation.
• ✍️ Narrative Formatter System - Technical specs for the Llama 3.2 conversational layer.
• 🔊 Multimodal 8GB Scaling - Roadmap for adding Audio & Text on limited hardware.
• 📞 Voice Calling Microservice - NEW Complete documentation for WebRTC telehealth with AI transcription, clinical reasoning, and bidirectional voice.

(For detailed breakdowns of dataset.py, dashboard.py, run_all_training.bat, etc., browse the docs/ directory).

---

💾 Supabase Setup (Persistence)

MedIQ uses Supabase for persistent conversation history and image storage.

1. Database Migration: Run the SQL scripts found in supabase/migrations/001_core_hms_schema.sql and 002_hms_rls_and_triggers.sql within your Supabase project's SQL Editor (Run as postgres admin).
2. Environment Variables: Update config/.env with your project credentials:

   SUPABASE_URL=https://your-project.supabase.co
   SUPABASE_SERVICE_KEY=your-service-role-key
   

3. Storage Buckets: Create a private bucket named mediq-images in the Supabase Storage dashboard.

---

🤖 The Foundation Models

This core visual system utilizes three foundational architectures, each uniquely fine-tuned or engineered to conquer the severe domain gap of clinical skin photos:

1. SigLIP2 (so400m-patch14): A sigmoid-loss-based model by Google offering dense semantic embeddings for clinical reasoning. The primary heavyweight identifier.
2. EVA-CLIP (EVA02-B-14): A highly compressed, high-performance model acting as our secondary diagnostic engine and OOD Condition-Gate evaluator.
3. OpenAI-CLIP (ViT-B-16): Maintained purely with un-finetuned "world knowledge" zero-shot weights to power the Stage 1 SkinGate filter.

---

⚙️ Core Architecture Features

• Soft & Hard Hierarchical Locking: Mentally splits 241 diseases into 16 broad families. Dynamically locks out mathematically impossible predictions to halt cross-family misdiagnosis.
• Retrieval-Augmented Generation (RAG): AI visual systems are not trusted to invent symptoms. They output a rigid diagnosis key (e.g. acne_vulgaris), which fetches human-verified facts from the clinical_kb.json.
• Narrative Formatter (Llama 3.2 3B): The local LLM (running via your Ollama server endpoint) is restricted exclusively to taking the deterministic JSON facts and rewriting them into a safe, patient-friendly bedside tone using aggressive loop-detection guardrails and template fallbacks.
• Async Non-Blocking Architecture: The Gateway uses a fully asynchronous lifecycle (acreate_client) with a tuned HTTP connection pool. This enables the system to handle thousands of concurrent guest sessions and medical reports without event-loop blocking.
• Test-Time Augmentation (TTA): The API dynamically sharpens, darkens, and flips submitted photos 5 times before averaging the neural consensus.

---

🛡️ High Availability & Scaling

MedIQ is engineered for production-grade resilience and horizontal scalability:

• Production Orchestration: Hardened docker-compose.yml with healthchecks, restart policies, and service_healthy readiness gates.
• Resource Management: Strict CPU and Memory limits (Docker cgroups) prevent container OOMs or compute starvation on constrained hardware.
• Connection Management: Conservative HTTP pooling (40 max connections) prevents "connection storms" from saturating the Supabase PostgREST backend while ensuring high TCP socket reuse.
• Stateless Gateway: The Gateway service stores absolutely no local state, allowing it to be scaled horizontally behind a load balancer with no session-stickiness requirements.

---

🚀 Quick Launch (Inference)

The API is fully isolated into Docker Microservices (one container per AI model to eliminate transformers dependency contamination).

# Bring up the Gateway and the active ensemble
docker compose up --build -d

# OR use the deployment script (includes Voice Service)
./deploy_multimodal.bat

The gateway exposes a unified orchestrator port (8003 by default) covering:

• POST /predict (Submit image. Executes Stage 1 & Stage 2 OOD Gates, then routes to single model or ?model=ensemble)
• POST /<model>/chat (Retrieve context-aware medical followups)
• POST /call/initiate (Start WebRTC telehealth consultation with AI)
• WS /ws/call/{call_id} (Real-time transcription + AI insights during call)
• (Direct backwards-compatible endpoints exist at /<model>/predict bypassing gates)

---

📞 Voice Calling (NEW)

MedIQ now supports real-time AI telehealth consultations with bidirectional voice:

Features

• 🎤 Real-time AI transcription during calls (Whisper)
• 🧠 Clinical reasoning - symptom extraction, red flag detection (Ollama)
• 🔊 AI doctor voice responses - patients can listen to AI (Kokoro TTS)
• 🌍 Bilingual support - English + French
• ⚡ Barge-in detection - patients can interrupt AI mid-sentence
• 🚨 Red flag alerts - immediate notifications during emergencies
• 📱 Mobile support - TURN server for NAT traversal

100% Local Execution

All AI models run locally with zero external API calls:

• Whisper (Speech-to-Text)
• Ollama (LLM Clinical Reasoning)
• Kokoro TTS (Text-to-Speech, ~500MB models)

Quick Start

# 1. Run database migration (manual)
supabase db push

# 2. Deploy voice service
./deploy_multimodal.bat

# 3. Test health
curl http://localhost:8010/health

# 4. Initiate call
curl -X POST http://localhost:8003/call/initiate \
  -H "Authorization: Bearer <token>" \
  -d "appointment_id=<uuid>"

📖 Complete Voice Calling Documentation

---

📸 Image Quality & Usage Guidelines

Because deep-learning vision models are highly sensitive to their input distribution, following these photographic guidelines is strictly required to pass the OOD (Out-of-Distribution) Gates and achieve maximum diagnostic accuracy:

1. Lighting & Clarity: Use bright, natural, or medical-grade white lighting. Avoid harsh shadows, extreme yellow indoor lighting, or overly dark photos.
2. Focus & Distance: The lesion/condition should be perfectly in focus and occupy at least 30-50% of the center frame. Avoid extreme macro shots that blur the edges, and avoid full-body shots where the lesion is a tiny speck.
3. Backgrounds: Try to keep backgrounds neutral (skin or a solid colored drape). Cluttered backgrounds (furniture, clothing patterns, food) risk triggering Stage 1 (SkinGate), which will cleanly reject the image as "non-skin".
4. Healthy Skin: Submitting photos of perfectly healthy, pristine skin will likely trigger Stage 2 (ConditionGate). The EVA-CLIP model actively scans for the presence of a pathological concept; if it cannot exceed its confidence threshold, the inference is halted to prevent hallucinated diagnoses.
5. Context: If a condition spans multiple body parts, take a clear representative shot of the most severe cluster rather than a stitched panoramic photo.

---

🎓 Training (Fine-Tuning)

The root workspace includes a master batch file designed to sequentially tune all three foundation models, applying optimal Epochs, MixUp, Focal Loss, and EMA regularization.

# Completely hands-free batch runner for all 5 foundation models
.\run_training.bat

Monitor live training progress, model health, and generalization gaps via the central hub:

venv\Scripts\streamlit run dashboard.py