🏥 VoicePilot: Enterprise Healthcare Audio-to-Claim Processing Pipeline

🎯 Overview

VoicePilot is a production-ready, enterprise-grade pipeline that converts healthcare audio recordings into structured claim data. Built with security, scalability, and maintainability in mind, it features comprehensive error handling, robust logging, and modular architecture.

✨ Key Features

• 🔒 Security-First Design: Environment-based configuration, no hardcoded credentials
• 📊 Comprehensive Monitoring: Detailed metrics, logging, and observability
• 🧪 Enterprise Testing: Full test coverage with pytest and quality checks
• 🐳 Production Ready: Docker support with security hardening
• 📚 Professional Documentation: Comprehensive docstrings and type hints
• 🔄 Modular Architecture: Clean separation of concerns and reusable components

🔄 Core Processing Stages

1. 🎵 Audio Cleaning: Advanced denoising and format normalization
2. 🗣️ Speech-to-Text (STT): High-accuracy Whisper-based transcription
3. 🤖 Claim Extraction: AI-powered structured data extraction using local LLM
4. 📡 API Integration: Secure transmission to external healthcare systems
5. 📊 Analytics & Monitoring: Comprehensive metrics and observability

🚀 Quick Start

Entry Points

• 🔄 End-to-End Pipeline: python main.py - Complete audio processing workflow
• 📡 API Integration: python app.py - Send extracted claims to external APIs
• 🧪 Development Tools: make help - View all available development commands

📁 Project Structure

VoicePilot/
├── 🚀 Entry Points
│   ├── app.py                     # API integration service
│   ├── main.py                    # End-to-end pipeline orchestrator
│   └── Makefile                   # Development commands
│
├── ⚙️ Configuration
│   ├── config_manager/
│   │   ├── config_pipeline.yaml   # Pipeline configuration
│   │   └── config_logging.yaml    # Logging settings
│   ├── env.example                # Environment variables template
│   └── pyproject.toml             # Project metadata & tool configs
│
├── 🔧 Core Processing
│   ├── scripts/
│   │   ├── audio_file_process/
│   │   │   ├── audio_cleaner.py   # Audio preprocessing
│   │   │   ├── speech_to_text.py  # Whisper transcription
│   │   │   ├── claim_extractor.py # AI claim extraction
│   │   │   ├── pipeline.py        # Orchestration & metrics
│   │   │   └── blob_storage_handler.py # Cloud storage
│   │   ├── API_Handler/
│   │   │   ├── api_handler.py     # API communication
│   │   │   └── api_server.py      # REST API server
│   │   ├── DB/
│   │   │   └── insert_audiofile.py # Database operations
│   │   └── dashboards/
│   │       └── dashboard.py       # Streamlit analytics
│   │
├── 🛠️ Utilities & Infrastructure
│   ├── utils/
│   │   ├── config_loader.py       # Configuration management
│   │   ├── logging_utils.py       # Structured logging
│   │   ├── constants.py           # Application constants
│   │   ├── validators.py          # Data validation
│   │   ├── exceptions.py          # Custom exceptions
│   │   ├── analytics.py           # Metrics & reporting
│   │   ├── pipeline_util.py       # Pipeline utilities
│   │   └── until_master.py        # Helper functions
│   │
├── 🧪 Testing & Quality
│   ├── tests/                     # Comprehensive test suite
│   ├── conftest.py                # Pytest configuration
│   ├── pytest.ini                # Test settings
│   └── .gitignore                 # Version control exclusions
│
├── 📊 Data & Logs
│   ├── local_data_source/         # Processing directories
│   ├── logs/                      # Rotating daily logs
│   └── metrics/                   # Performance metrics
│
├── 🐳 Deployment
│   ├── Dockerfile                 # Container configuration
│   ├── requirements.txt           # Python dependencies
│   └── run_pipeline.sh            # Setup script
│
└── 📚 Documentation
    └── README.md                  # This comprehensive guide

⚙️ Configuration & Setup

🔧 Configuration Files

Pipeline Configuration (config_manager/config_pipeline.yaml)

• 📁 Paths: Directory mappings for all processing stages
• 🎵 Audio: STT model settings and supported formats
• 🤖 AI: LLM configuration and prompt templates
• 🗄️ Database: Optional SQL Server connection settings
• 📊 Retention: Log and file cleanup policies

Logging Configuration (config_manager/config_logging.yaml)

• 📝 Structured Logging: Rotating daily logs with retention
• 📧 Email Alerts: Configurable notifications for critical errors
• 🔍 Log Levels: Runtime configurable via VOICLAIM_LOG_LEVEL

🔐 Environment Variables

⚠️ Security Note: All sensitive data is now managed via environment variables. Copy env.example to .env and configure your values.

Required Variables

# API Configuration
POST_PROCESS_URL=https://your-api-endpoint.com/process
CONTENT_TYPE=application/json
DEPLOYMENT_KEY=your-deployment-key
X_VA_SENDERAGENT_ID=your-sender-agent-id

# Database (if using SQL Server)
DB_PROD_HOST=your-db-host
DB_PROD_DATABASE=your-database
DB_PROD_USER=your-username
DB_PROD_PASSWORD=your-password

Optional Variables

# Logging
VOICLAIM_LOG_LEVEL=INFO  # DEBUG, INFO, WARNING, ERROR, CRITICAL

# API Timeout
API_TIMEOUT_SEC=30

# Processing
MAX_WORKERS=2

🚀 Getting Started

📋 Prerequisites

• Python 3.10+ with pip
• Virtual Environment (recommended)
• Docker (for containerized deployment)
• Ollama (for local LLM processing)

🛠️ Installation

Option 1: Quick Setup (Recommended)

# Clone and setup
git clone <repository-url>
cd VoicePilot
./run_pipeline.sh  # Automated setup script

Option 2: Manual Setup

# Create virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies
pip install --upgrade pip
pip install -r requirements.txt

# Setup environment
cp env.example .env
# Edit .env with your configuration

🏃‍♂️ Running the Pipeline

Development Commands

# View all available commands
make help

# Install dependencies
make install

# Run tests
make test

# Code quality checks
make lint
make format
make security

Production Usage

🔄 End-to-End Pipeline

python main.py \
  --max-workers 2 \
  --day 1 \
  --dry-run            # Skip API calls for testing

📡 API Integration Only

python app.py \
  --base_folder local_data_source/extracted_claims/ \
  --test_file local_data_source/extracted_claims/sample.json

🐳 Docker Deployment

# Build image
make docker-build

# Run container
make docker-run

📊 Data Flow

📁 local_data_source/
├── raw_audio/          # Input audio files
├── processing/         # In-flight processing
├── cleaned_audio/      # Preprocessed audio
├── transcripts/        # Speech-to-text output
├── extracted_claims/   # AI-extracted JSON claims
├── processed/          # Completed artifacts
├── failed/            # Error diagnostics
└── success/           # Success receipts

📊 Monitoring & Observability

📈 Metrics & Analytics

• 📁 Comprehensive Metrics: Detailed CSV reports in metrics/ directory
• 🔄 Pipeline Lifecycle: End-to-end processing tracking
• ⚡ Performance Monitoring: Stage-level timing and resource usage
• 📊 Success/Failure Rates: Automated quality metrics

📝 Logging & Debugging

• 📅 Rotating Logs: Daily log files with automatic retention
• 🔍 Structured Logging: JSON-formatted logs for easy parsing
• 📧 Alert System: Configurable email notifications for critical errors
• 🎛️ Runtime Control: Dynamic log level adjustment via environment variables

🗄️ Database Integration

• 🔗 SQL Server Support: Optional database logging and tracking
• 📊 Data Consistency: UUID-based record linking across tables
• 🔐 Secure Configuration: Environment-based credential management
• 📈 Audit Trail: Complete processing history and outcomes

🧪 Testing & Quality Assurance

🧪 Test Suite

# Run all tests
make test

# Run with coverage
make test-coverage

# Run specific test categories
pytest tests/ -m unit
pytest tests/ -m integration

🔍 Code Quality

# Format code
make format

# Lint code
make lint

# Security scan
make security

# Type checking
make type-check

📊 Quality Metrics

• ✅ 100% Test Pass Rate: All tests passing
• 🔒 Security Scanned: Bandit security analysis
• 📝 Type Hints: Comprehensive type annotations
• 🎨 Code Formatted: Black formatting applied
• 📚 Documented: Full docstring coverage

🐳 Production Deployment

🐳 Docker Support

• 🔒 Security Hardened: Non-root user, minimal attack surface
• 📦 Self-Contained: All dependencies included
• ⚡ Optimized: Multi-stage build for smaller images
• 🔄 Health Checks: Built-in container health monitoring

☁️ Cloud Deployment

• 🌐 Container Ready: Docker and Kubernetes compatible
• 📊 Monitoring: Prometheus metrics and Grafana dashboards
• 🔐 Secrets Management: Integration with cloud secret managers
• 📈 Auto-Scaling: Horizontal scaling support

🛡️ Security & Compliance

🔐 Security Features

• 🚫 No Hardcoded Secrets: All credentials via environment variables
• 🔍 Input Validation: Comprehensive data sanitization
• 📝 Audit Logging: Complete processing audit trail
• 🛡️ Error Handling: Secure error messages without data leakage

📋 Compliance

• 🏥 Healthcare Ready: HIPAA-compliant data handling
• 🔒 Data Privacy: PII masking and secure processing
• 📊 Audit Trail: Complete processing history
• 🛡️ Access Control: Role-based access patterns

🤝 Contributing

🛠️ Development Setup

# Clone repository
git clone <repository-url>
cd VoicePilot

# Setup development environment
make install-dev

# Run pre-commit checks
make pre-commit

📝 Code Standards

• 🎨 Black Formatting: Consistent code style
• 📚 Docstrings: Comprehensive function documentation
• 🧪 Tests: Unit and integration test coverage
• 🔍 Type Hints: Full type annotation coverage

📞 Support & Documentation

📚 Additional Resources

• 🔧 Configuration Guide: Detailed setup instructions
• 🐛 Troubleshooting: Common issues and solutions
• 📊 Performance Tuning: Optimization recommendations
• 🔐 Security Best Practices: Deployment security guide

🆘 Getting Help

• 📧 Issues: GitHub Issues for bug reports
• 💬 Discussions: GitHub Discussions for questions
• 📖 Wiki: Comprehensive documentation wiki
• 🎥 Tutorials: Step-by-step video guides

---

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• OpenAI Whisper for speech-to-text capabilities
• Ollama for local LLM processing
• FastAPI for high-performance web framework
• Streamlit for interactive dashboards