🏥 IVF Clinical Document Digitization Platform

---

📋 Table of Contents

• Overview
• Key Features
• Technical Architecture
• Technology Stack
• Installation & Setup
• Usage Guide
• System Components
• Data Model
• Known Limitations
• Future Roadmap
• Contributing
• Acknowledgments

---

🎯 Overview

The IVF Clinical Document Digitization Platform is a comprehensive web-based solution designed to transform unstructured IVF (In Vitro Fertilization) medical records into structured, machine-readable datasets. This platform bridges the gap between traditional paper-based clinical documentation and modern AI-driven healthcare systems.

The Problem

IVF clinics generate vast amounts of clinical data in PDF and image formats, including:

• Patient demographic information
• Hormonal assay results (AMH, FSH, Estradiol)
• Ultrasound measurements (AFC, Endometrium thickness)
• Embryo grading reports
• Medication protocols and dosages
• Clinical notes and observations

This data remains locked in unstructured formats, making it:

• ❌ Difficult to analyze at scale
• ❌ Impossible to use for AI/ML model training
• ❌ Time-consuming to extract manually
• ❌ Prone to transcription errors

The Solution

Our platform provides an expert-in-the-loop system that:

• ✅ Automatically extracts text from medical documents using OCR
• ✅ Presents interactive bounding boxes for text verification and correction
• ✅ Enables clinical experts to label extracted data with IVF-specific field types
• ✅ Generates structured annotations suitable for AI training datasets
• ✅ Maintains data quality through human oversight and validation

---

✨ Key Features

🔍 Intelligent OCR Processing

• Multi-format Support: Handles PDF documents and image files (PNG, JPG, TIFF)
• Page-by-Page Processing: Efficiently processes multi-page documents with individual page navigation
• Automatic Text Detection: Leverages Tesseract OCR engine for robust text extraction
• Bounding Box Generation: Automatically creates interactive bounding boxes around detected text regions

🎨 Modern, Medical-Themed UI

• Professional Design: Warm color palette with teal gradients (#6dd5b9 to #4db8a0) and soft medical colors
• Institutional Branding: Integrated Ashoka University and Kutum Labs logos
• Responsive Layout: Two-panel interface with document viewer and annotation panel
• Intuitive Status Bar: Real-time feedback on processing status and user actions

📦 Interactive Bounding Box Management

• Transparent Overlays: Subtle bounding boxes that don't obscure document text
• Hover-Activated Controls: Controls appear only when needed, maintaining visual clarity
• Drag-to-Move: Click and drag boxes to reposition them over correct text regions
• Resize Handles: Drag corner handles to adjust box dimensions
• Double-Click Editing: Quick text editing via double-click on box labels
• Delete Functionality: Remove incorrect or unnecessary boxes with a single click
• Visual Highlighting: Selected boxes highlight with distinct borders for easy identification

🏷️ Clinical Field Annotation System

• IVF-Specific Labels: Predefined dropdown with 12 clinical field types:
  • Patient Name
  • Age
  • AMH (Anti-Müllerian Hormone)
  • AFC (Antral Follicle Count)
  • Cycle Day
  • Estradiol
  • FSH (Follicle Stimulating Hormone)
  • Endometrium Thickness
  • Embryo Grade
  • Medication Name
  • Dose
  • Notes
• Color-Coded Labels: Each field type has a unique background color for visual organization
• Annotation List: Real-time display of all saved annotations with page numbers and text previews
• Manual Text Selection: Ability to select and annotate any text from OCR output
• Flexible Classification: Option for manual input or custom labels

🛠️ Advanced Editing Capabilities

• Add New Boxes: Create bounding boxes centered on the page or aligned to selected words
• Update Box Content: Modify text and class labels for existing boxes
• Batch Operations: Work with multiple annotations across different pages
• Undo-Friendly: Delete unwanted annotations from the annotation list

💾 Data Persistence

• Backend API Integration: Save annotations to Flask backend via RESTful API
• Structured JSON Output: Exports data in standardized format for downstream AI training
• File Association: Links annotations to original source documents
• Page Tracking: Maintains page numbers and relative bounding box coordinates

---

🏗️ Technical Architecture

System Design

┌─────────────────────────────────────────────────────────────┐
│                        Frontend (React)                      │
│  ┌────────────────┐  ┌──────────────┐  ┌─────────────────┐ │
│  │  File Upload   │  │  PDF Viewer  │  │ Annotation Panel│ │
│  └────────┬───────┘  └──────┬───────┘  └────────┬────────┘ │
│           │                  │                    │          │
│           └──────────────────┼────────────────────┘          │
│                              │                               │
│                    ┌─────────▼──────────┐                    │
│                    │  BoundingOverlay   │                    │
│                    │   Component        │                    │
│                    └────────────────────┘                    │
└──────────────────────────────┬──────────────────────────────┘
                               │ HTTP/JSON
                               │
┌──────────────────────────────▼──────────────────────────────┐
│                     Backend (Flask)                          │
│  ┌────────────────┐  ┌──────────────┐  ┌─────────────────┐ │
│  │ OCR Inference  │  │ Annotation   │  │  Data Storage   │ │
│  │  (Tesseract)   │  │   Endpoint   │  │   (JSON/YAML)   │ │
│  └────────────────┘  └──────────────┘  └─────────────────┘ │
└─────────────────────────────────────────────────────────────┘

Component Hierarchy

App.jsx (Main Application)
├── Header.jsx (Banner with logos and title)
├── StatusBar.jsx (Status messages)
├── FileUploadButton (Custom file input)
├── Document Viewer
│   ├── React-PDF (PDF rendering)
│   ├── Image Preview (Image rendering)
│   └── BoundingOverlay.jsx (Interactive boxes)
├── OCR Text Panel
│   └── ClickableText.jsx (Selectable text)
└── Annotation Panel
    ├── Selected Text Display
    ├── Clinical Field Dropdown
    ├── Annotation List
    └── Bounding Box Editor

Data Flow

1. Document Upload → User selects PDF/image file
2. OCR Processing → Backend extracts text with Tesseract and returns JSON with text, bounding boxes, and page info
3. Visualization → Frontend renders document with transparent bounding box overlays
4. User Interaction → Expert clicks/drags boxes, selects text, assigns labels
5. Annotation Creation → System creates structured annotation objects with text, class, page, and bbox coordinates
6. Data Export → Annotations saved to backend in JSON format for AI training

---

💻 Technology Stack

Frontend

Backend

Development Tools

• Node.js & npm: JavaScript runtime and package manager
• Python 3.8+: Backend runtime
• Git: Version control
• VS Code: Recommended IDE

---

🚀 Installation & Setup

Prerequisites

# System Requirements
- Node.js 16+ and npm
- Python 3.8+
- Tesseract OCR 4.0+
- Poppler (for PDF processing)

Step 1: Clone Repository

git clone https://github.com/anuragmoyde/AIIMS-IVF-Capstone.git
cd AIIMS-IVF-Capstone

Step 2: Install System Dependencies

macOS:

brew install tesseract
brew install poppler

Ubuntu/Debian:

sudo apt-get update
sudo apt-get install tesseract-ocr
sudo apt-get install poppler-utils

Windows:

• Download Tesseract installer from GitHub
• Download Poppler from oschwartz10612/poppler-windows

Step 3: Backend Setup

cd app/backend

# Create virtual environment
python -m venv .venv

# Activate virtual environment
# macOS/Linux:
source .venv/bin/activate
# Windows:
.venv\Scripts\activate

# Install Python dependencies
pip install -r requirements.txt

Step 4: Frontend Setup

cd app/frontend

# Install Node dependencies
npm install

Step 5: Add Institutional Logos

Place logo files in app/frontend/src/assets/:

• ashoka logo.jpg (Ashoka University logo)
• kutum-labs-logo.png (Kutum Labs logo)

---

🎮 Usage Guide

Starting the Application

Terminal 1 - Backend Server:

cd app/backend
source .venv/bin/activate  # Activate virtual environment
python main.py
# Server runs on http://localhost:5050

Terminal 2 - Frontend Development Server:

cd app/frontend
npm run dev
# Application opens at http://localhost:5173 (or specified port)

Workflow

1️⃣ Upload Document

• Click "Upload File" button
• Select a PDF or image file (PNG, JPG, TIFF)
• Preview appears in left panel

2️⃣ Run OCR

• Click "Run OCR" button
• Wait for processing (5-10 seconds)
• Text extraction completes with bounding boxes overlaid

3️⃣ Review & Correct Bounding Boxes

• Hover over boxes to see controls
• Click a box to select it
• Drag to reposition
• Drag corner handle to resize
• Double-click label to edit text
• Click × to delete incorrect boxes

4️⃣ Annotate Clinical Fields

• Select text from OCR output (or click a box)
• Choose IVF Clinical Field from dropdown
• Click "Save Label" to add annotation
• View in Annotation List with color coding

5️⃣ Add Custom Boxes

• Enter text in "Box text" field
• Optionally add class in "Class (optional)"
• Click "Add Box (center)" or "Align to Selected"

6️⃣ Navigate Multi-Page Documents

• Use Previous / Next buttons
• Track progress with page counter

7️⃣ Save Annotations

• Click "Save Annotations" button
• Data exports to backend in JSON format
• Ready for AI model training

---

🧩 System Components

Frontend Components

App.jsx (Main Application Controller)

• State Management: Manages file state, OCR results, annotations, and UI state
• API Integration: Communicates with Flask backend via Axios
• Event Handlers: Processes user interactions (clicks, drags, text selection)
• Rendering Logic: Orchestrates all child components

Key Functions:

• handleFileChange(): Processes uploaded files
• runOCR(): Sends files to backend for OCR processing
• handleUpdateBox(): Updates bounding box properties
• handleDeleteBox(): Removes bounding boxes
• addAnnotationFromSelection(): Creates structured annotations
• saveAnnotationsToBackend(): Exports data to Flask API

BoundingOverlay.jsx (Interactive Box Layer)

• Dynamic Rendering: Creates draggable, resizable boxes from relative coordinates
• Mouse Event Handling: Implements drag-to-move and resize functionality
• Visual Feedback: Hover states, highlighting, and control visibility
• Coordinate System: Converts between relative (0-1) and absolute pixel coordinates

Features:

• Transparent backgrounds (rgba(109, 213, 185, 0.08))
• Teal borders that intensify on hover/selection
• Positioned resize handles
• Delete buttons with gradient styling

ClickableText.jsx (Selectable OCR Output)

• Text Rendering: Displays extracted OCR text
• Word-Level Interaction: Makes individual words clickable
• Annotation Highlighting: Color-codes annotated text
• Selection Handling: Captures user text selections

Header.jsx (Platform Banner)

• Branding: Displays institutional logos
• Title & Description: Platform name and mission statement
• Gradient Background: Medical-themed color scheme

StatusBar.jsx (Status Display)

• Real-Time Feedback: Shows current operation status
• User Guidance: Provides contextual messages

Backend Components

main.py (Flask Application)

• OCR Endpoint (/inference): Processes uploaded files with Tesseract
• Annotation Endpoint (/save_annotation): Saves structured annotations
• CORS Configuration: Enables frontend-backend communication
• Error Handling: Graceful error responses

API Endpoints:

POST /inference

Request: multipart/form-data with "image" field
Response: {
  "pages": [
    {
      "page": 1,
      "text": "extracted text...",
      "words": [
        {
          "text": "Patient",
          "bbox_rel": {"x": 0.1, "y": 0.2, "w": 0.15, "h": 0.03},
          "class": null,
          "source": "ocr"
        }
      ]
    }
  ]
}

POST /save_annotation

Request: {
  "file_id": "filename.pdf",
  "annotations": [
    {
      "text": "John Doe",
      "class": "Patient Name",
      "page": 1,
      "bbox_rel": {"x": 0.25, "y": 0.2, "w": 0.2, "h": 0.03},
      "source": "pdf"
    }
  ]
}
Response: {"status": "success"}

annotation_schema.yaml (LinkML Schema)

• Data Validation: Defines annotation structure
• Type Constraints: Ensures data integrity
• Documentation: Schema for future extensions

---

📊 Data Model

Annotation Object Structure

interface Annotation {
  text: string;              // Extracted or edited text
  class: string | null;      // IVF clinical field type
  page: number;              // Source page number
  bbox_rel: BBox;           // Relative bounding box coordinates
  source: "ocr" | "pdf" | "human";  // Origin of annotation
}

interface BBox {
  x: number;    // Left position (0-1)
  y: number;    // Top position (0-1)
  w: number;    // Width (0-1)
  h: number;    // Height (0-1)
}

Clinical Field Types

1. Patient Name - Full name of IVF patient
2. Age - Patient age at treatment time
3. AMH - Anti-Müllerian Hormone level (ng/mL)
4. AFC - Antral Follicle Count
5. Cycle Day - Day of menstrual cycle
6. Estradiol - Estrogen level (pg/mL)
7. FSH - Follicle Stimulating Hormone (mIU/mL)
8. Endometrium Thickness - Uterine lining measurement (mm)
9. Embryo Grade - Embryo quality classification
10. Medication Name - Prescribed fertility medication
11. Dose - Medication dosage
12. Notes - Clinical observations

---

⚠️ Known Limitations

🚫 Current Constraints

Handwritten Document Support

❌ CRITICAL LIMITATION: The platform currently does not support handwritten clinical documents.

Technical Reason: Tesseract OCR, while excellent for printed text, has limited accuracy with handwritten content, especially:

• Cursive or connected handwriting
• Variable handwriting styles across different clinicians
• Medical shorthand and abbreviations
• Overlapping text and corrections

Workaround: For handwritten documents, consider:

• Manual transcription by clinical staff
• Using specialized handwriting recognition models (e.g., Google Cloud Vision API, Microsoft Azure Computer Vision)
• Hybrid approach: OCR for printed sections, manual entry for handwritten notes

Language Support

• Currently optimized for English language documents
• Limited support for documents with mixed languages
• Requires Tesseract language packs for non-English text

Document Quality

• Best Results: High-resolution scans (300+ DPI), clear printed text, good contrast
• Poor Results: Low-quality scans, faded text, background noise, skewed pages
• Recommendation: Use document scanning best practices (proper lighting, flat surface, high resolution)

Performance Considerations

• Large multi-page PDFs (50+ pages) may take extended processing time
• Memory-intensive for high-resolution images
• Single-threaded OCR processing (no parallel page processing)

Browser Compatibility

• Optimized for modern browsers (Chrome 90+, Firefox 88+, Safari 14+)
• Limited support for Internet Explorer
• Requires JavaScript enabled

---

🗺️ Future Roadmap

Short-Term Goals (Q1 2025)

• Handwriting Recognition Integration: Evaluate and integrate specialized handwriting OCR models
• Batch Processing: Enable simultaneous processing of multiple documents
• Export Formats: Add CSV, Excel, and FHIR-compliant JSON exports
• User Authentication: Implement role-based access control
• Audit Logging: Track all annotation changes for quality assurance

Medium-Term Goals (Q2-Q3 2025)

• Alternative OCR Engines: Integrate PaddleOCR, EasyOCR, and VLM-based solutions
• Performance Comparison Dashboard: Real-time accuracy metrics across OCR engines
• Auto-Classification: ML model to predict clinical field types
• Annotation Templates: Predefined templates for common IVF document types
• Collaborative Annotation: Multi-user support with conflict resolution

Long-Term Goals (Q4 2025 & Beyond)

• Mobile Application: iOS/Android apps for on-the-go annotation
• Cloud Deployment: Production-ready deployment on AWS/Azure/GCP
• AI-Assisted Correction: Active learning to suggest corrections
• Integration with EMR Systems: Direct export to electronic medical records
• Multilingual Support: Expand to support regional languages

---

🤝 Contributing

We welcome contributions from the community! Here's how you can help:

Areas for Contribution

• OCR Engine Improvements: Test and integrate alternative OCR solutions
• UI/UX Enhancements: Improve design and user experience
• Documentation: Expand guides, tutorials, and API documentation
• Testing: Write unit tests, integration tests, and end-to-end tests
• Bug Fixes: Report and fix issues
• Feature Development: Implement items from the roadmap

Development Guidelines

1. Fork the repository
2. Create a feature branch (git checkout -b feature/AmazingFeature)
3. Commit changes (git commit -m 'Add AmazingFeature')
4. Push to branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

Code Standards

• Frontend: Follow React best practices, ESLint configuration
• Backend: PEP 8 style guide for Python code
• Commits: Use conventional commit messages
• Testing: Include tests for new features

---

📜 License

This project is developed for research purposes in collaboration with:

• Ashoka University - Academic research institution
• Kutum Labs - Healthcare AI research lab

Note: License details to be determined based on institutional requirements.

---

🙏 Acknowledgments

Institutions

• Ashoka University - For academic support and research guidance
• Kutum Labs - For technical expertise and domain knowledge
• AIIMS (All India Institute of Medical Sciences) - For providing real-world IVF clinical data

Technologies

• Tesseract OCR - Open-source OCR engine by Google
• React Community - For excellent frontend libraries
• Flask Community - For lightweight backend framework

Contributors

• Development Team at Anurag Moyde
• Clinical experts who provided domain knowledge
• Open-source contributors

---

📞 Contact & Support

Project Maintainer

• Name: Anurag Moyde
• GitHub: @anuragmoyde
• Repository: AIIMS-IVF-Capstone

Reporting Issues

Please report bugs and feature requests via GitHub Issues.

Documentation

For additional documentation, see the /docs folder (to be added).

---