Universal Multimodal RAG Platform

A production-grade AI knowledge assistant that supports multimodal data ingestion, processing, and conversational query across all uploaded content.

🚀 Features

Core Capabilities

• Multimodal Data Processing: Support for documents, images, audio, video, and structured data
• Intelligent RAG: Advanced retrieval with semantic chunking and reranking
• Voice Interaction: Speech-to-text and text-to-speech capabilities
• Agent Orchestration: LangChain-powered reasoning with dynamic tool selection
• Production-Ready: Scalable architecture with monitoring and safety layers

Supported File Types

• Documents: PDF, DOCX, TXT, RTF
• Images: PNG, JPG, JPEG, GIF, BMP, TIFF, WebP (with OCR)
• Audio: WAV, MP3, OGG, FLAC, M4A (with transcription)
• Video: MP4, MOV, AVI, MKV, WebM (with keyframe extraction)
• Structured Data: CSV, Excel, JSON

AI Features

• Semantic Search: Advanced vector similarity search with Qdrant
• Intelligent Chunking: Context-aware text segmentation
• Safety Layers: Prompt injection protection and content filtering
• Multi-Model Support: NVIDIA Nemotron, Qwen2.5, Mixtral, DeepSeek
• Real-time Processing: Streaming responses and async background processing

🏗️ Architecture

System Components

┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐
│   Frontend      │  │    Backend      │  │   Qdrant       │
│   (React)       │  │   (FastAPI)     │  │   Vector DB    │
└─────────────────┘  └─────────────────┘  └─────────────────┘
         │                   │                   │
         └───────────────────┼───────────────────┘
                             │
                    ┌─────────────────┐
                    │   GPU Services  │
                    │   (LLM/Embed)   │
                    └─────────────────┘

Technology Stack

• Backend: FastAPI, Python 3.11
• Frontend: React 18, TypeScript, Vite
• Vector Database: Qdrant
• AI Models: NVIDIA Nemotron, BGE-M3, Sentence Transformers
• Infrastructure: Docker, Docker Compose, Nginx
• Monitoring: Prometheus, Grafana, Structured Logging

🛠️ Installation

Prerequisites

• Docker and Docker Compose
• NVIDIA GPU (optional, for accelerated inference)
• NVIDIA API Key (for Nemotron models)

Quick Start

1. Clone the repository

git clone <repository-url>
cd Multi-Universal-RAG

2. Configure environment

cp .env.example .env
# Edit .env with your NVIDIA_API_KEY and other settings

3. Deploy the platform

chmod +x deploy.sh
./deploy.sh

4. Access the application

• Frontend: http://localhost:3000
• Backend API: http://localhost:8000
• API Documentation: http://localhost:8000/docs

Manual Deployment

# Start all services
docker-compose up -d

# View logs
docker-compose logs -f

# Stop services
docker-compose down

📖 Usage

1. Upload Datasets

1. Navigate to the Datasets page
2. Create or select a collection
3. Drag and drop files or use the upload button
4. Monitor processing progress
5. View detailed statistics once completed

2. Chat with Your Data

1. Go to the Chat interface
2. Use voice input or type your question
3. Toggle between simple RAG and agent mode
4. View responses with source citations
5. Upload files directly in chat for immediate processing

3. Advanced Features

• Semantic Search: Direct vector database exploration
• Collection Management: Organize datasets into knowledge bases
• Voice Interaction: Full speech-to-text and text-to-speech
• Image Analysis: Upload images for VLM analysis
• Real-time Monitoring: Track system performance and usage

🔧 Configuration

Environment Variables

# NVIDIA API
NVIDIA_API_KEY=your_api_key_here

# Vector Database
QDRANT_HOST=localhost
QDRANT_PORT=6333

# Application
ENVIRONMENT=production
LOG_LEVEL=INFO

# Models
EMBEDDING_MODEL=BAAI/bge-m3
RERANKER_MODEL=BAAI/bge-reranker-base
LLM_MODEL=nvidia/nemotron-4-340b-instruct

Model Configuration

• Embeddings: BGE-M3 (default), E5-Mistral, NVIDIA Nemotron
• Reranking: BGE Reranker, NVIDIA Nemotron Reranker
• LLM: NVIDIA Nemotron, Qwen2.5, Mixtral, DeepSeek

📊 Monitoring

System Metrics

• Performance: Response times, throughput, error rates
• Resource: CPU, memory, disk usage
• Application: Active connections, request counts
• AI Operations: Embedding, vector, processing metrics

Access Points

• Prometheus: http://localhost:9090
• Grafana: http://localhost:3001 (admin/admin)
• Health Checks: http://localhost:8000/health/detailed

🔒 Security

Safety Features

• Prompt Injection Protection: Pattern-based detection
• Content Filtering: Harmful content detection
• PII Protection: Automatic redaction of sensitive information
• Input Validation: Comprehensive input sanitization
• Rate Limiting: API abuse prevention

Monitoring

• Security Events: Logged and alerted
• Access Control: JWT-based authentication (configurable)
• Audit Trails: Complete request/response logging

🚀 Performance

Optimizations

• Semantic Chunking: Intelligent text segmentation
• Embedding Caching: Avoid recomputation
• Batch Processing: Efficient background operations
• Async Operations: Non-blocking I/O throughout
• Streaming Responses: Real-time user experience

Scalability

• Horizontal Scaling: Multi-container deployment
• Load Balancing: Nginx reverse proxy
• Caching Layers: Redis for session and response caching
• Database Clustering: Qdrant for large datasets

📚 API Documentation

Core Endpoints

# Collections
POST /collections          # Create collection
GET  /collections          # List collections
GET  /collections/{id}     # Get collection
DELETE /collections/{id}  # Delete collection

# Datasets
POST /datasets/upload      # Upload dataset
GET  /datasets            # List datasets
GET  /datasets/{id}       # Get dataset
DELETE /datasets/{id}    # Delete dataset

# Query
POST /query               # Text query with RAG
POST /voice-query          # Voice query
POST /search               # Semantic search

# System
GET  /health              # Basic health check
GET  /health/detailed     # Detailed health with metrics
GET  /metrics             # Prometheus metrics
GET  /statistics          # System statistics

Example Usage

import requests

# Upload a dataset
files = {'file': open('document.pdf', 'rb')}
data = {'collection_id': 'your-collection-id'}
response = requests.post('http://localhost:8000/datasets/upload', files=files, data=data)

# Query your data
query_data = {
    'input_text': 'What are the key findings in the uploaded documents?',
    'use_agent': True,
    'top_k': 5
}
response = requests.post('http://localhost:8000/query', json=query_data)
print(response.json())

🧪 Development

Local Development Setup

# Backend
cd backend
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -r requirements.txt
uvicorn app:app --reload

# Frontend
cd frontend
npm install
npm run dev

Testing

# Backend tests
cd backend
pytest

# Frontend tests
cd frontend
npm test

🔧 Troubleshooting

Common Issues

Q: GPU not detected? A: Ensure NVIDIA drivers and Docker GPU support are installed. Uncomment GPU sections in docker-compose.yml.

Q: Uploads failing? A: Check file size limits and disk space. Verify NVIDIA_API_KEY is correctly set.

Q: Slow responses? A: Monitor system resources. Consider scaling or using smaller models for faster inference.

Q: Memory issues? A: Reduce batch sizes, enable streaming, or scale horizontally.

Logs

# View all logs
docker-compose logs -f

# View specific service
docker-compose logs -f backend
docker-compose logs -f frontend
docker-compose logs -f qdrant

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests if applicable
5. Submit a pull request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• NVIDIA: For Nemotron models and API access
• Qdrant Team: For the vector database
• LangChain: For agent orchestration
• Hugging Face: For open-source models
• FastAPI: For the web framework

📞 Support

For support and questions:

• Create an issue in the repository
• Check the documentation at /docs
• Monitor system health at /health/detailed

---

Universal Multimodal RAG Platform - Transform your data into intelligent conversations. 🚀