🎯 What is RAG Modulo?

RAG Modulo is a production-ready Retrieval-Augmented Generation platform that provides enterprise-grade document processing, intelligent search, and AI-powered question answering with advanced Chain of Thought (CoT) reasoning. Built with modern technologies and designed for scalability, it supports multiple vector databases (Milvus, Elasticsearch, Pinecone, Weaviate, ChromaDB), LLM providers (WatsonX, OpenAI, Anthropic), and document formats including enhanced support via IBM Docling integration.

✨ Key Features

🎨 Frontend Features

• Modern UI: React 18 with Tailwind CSS and Carbon Design System principles
• Reusable Component Library: 8 accessible, type-safe components with consistent design patterns
• Enhanced Search: Interactive chat interface with Chain of Thought reasoning visualization
• Document Management: Real-time file upload with drag-and-drop support and validation
• Smart Display: Document source attribution with chunk-level page references
• Podcast Generation: AI-powered podcast creation with voice preview
• Question Suggestions: Intelligent query recommendations based on collection content

🎉 Current Status: Production Ready

🎉 Recent Major Improvements

🚀 Quick Start

Prerequisites

📋 Verify Installation: Run make check-docker to verify Docker requirements

Option 1: Local Development (⚡ Fastest - Recommended)

Best for: Daily development, feature work, rapid iteration

# 1. Clone repository
git clone https://github.com/manavgup/rag_modulo.git
cd rag_modulo

# 2. Set up environment variables
cp .env.example .env
# Edit .env with your API keys (WatsonX, OpenAI, Anthropic)

# 3. Install all dependencies (one command!)
make local-dev-setup
# This creates Python venv + installs backend (Poetry) + frontend (npm)

# 4. Start infrastructure only (Postgres, Milvus, MinIO, MLFlow)
make local-dev-infra

# 5a. Option A: Start backend and frontend in separate terminals
# Terminal 1:
make local-dev-backend

# Terminal 2:
make local-dev-frontend

# 5b. Option B: Start everything in background (easier!)
make local-dev-all

Access Points:

• 🌐 Frontend: http://localhost:3000
• 🔧 Backend API: http://localhost:8000/docs (Swagger UI)
• 📊 MLFlow: http://localhost:5001
• 💾 MinIO Console: http://localhost:9001

Benefits:

• ⚡ Instant reload - Python/React changes reflected immediately (no container rebuilds)
• 🐛 Native debugging - Use PyCharm, VS Code debugger with breakpoints
• 📦 Local caching - Poetry/npm caches work natively for faster dependency installs
• 🔥 Fastest iteration - Pre-commit hooks optimized (fast on commit, comprehensive on push)

When to use:

• ✅ Daily development work
• ✅ Feature development and bug fixes
• ✅ Rapid iteration and testing
• ✅ Debugging with breakpoints

Option 2: Production Mode (🐳 Docker)

Best for: Production-like testing, deployment validation

# Clone repository
git clone https://github.com/manavgup/rag_modulo.git
cd rag_modulo

# Set up environment
cp .env.example .env
# Edit .env with your API keys

# Build production images and start
make build-all
make prod-start

When to use:

• ✅ Testing production configurations
• ✅ Validating Docker builds
• ✅ Deployment rehearsal
• ✅ Performance benchmarking

Option 3: GitHub Codespaces (☁️ Cloud)

Best for: Quick experimentation, onboarding, cloud development

1. Go to repository → "Code" → "Codespaces"
2. Click "Create codespace" on your branch
3. Start coding in browser-based VS Code
4. Run: make venv && make local-dev-infra

When to use:

• ✅ No local setup required
• ✅ Consistent development environment
• ✅ Work from any device
• ✅ Team onboarding

🏗️ Architecture Overview

RAG Modulo follows a modern, service-based architecture with clear separation of concerns:

graph TB
    subgraph "Frontend Layer"
        UI[React Web UI]
        CLI[Command Line Interface]
    end

    subgraph "API Layer"
        API[FastAPI Backend]
        AUTH[OIDC Authentication]
    end

    subgraph "Service Layer"
        SEARCH[Search Service]
        CONV[Conversation Service]
        TOKEN[Token Tracking]
        COT[Chain of Thought]
    end

    subgraph "Data Layer"
        VDB[(Vector Database)]
        PG[(PostgreSQL)]
        MINIO[(MinIO Storage)]
    end

    subgraph "External Services"
        LLM[LLM Providers]
        EMB[Embedding Models]
    end

    UI --> API
    CLI --> API
    API --> SEARCH
    API --> CONV
    API --> TOKEN
    API --> COT
    SEARCH --> VDB
    SEARCH --> PG
    CONV --> LLM
    TOKEN --> PG
    COT --> LLM
    API --> MINIO

🛠️ Development Workflow

🎯 Recommended Daily Workflow

Philosophy: Develop locally without containers for maximum speed, use containers only for infrastructure and production.

# ============================================================
# ONE-TIME SETUP (First Day Only)
# ============================================================
make local-dev-setup    # Creates venv + installs all dependencies

# ============================================================
# DAILY ROUTINE (Every Day)
# ============================================================

# Option A: Manual Start (Recommended for debugging)
# ---------------------------------------------------
make local-dev-infra              # Start infrastructure

# Terminal 1: Backend
make local-dev-backend            # Auto-reload enabled

# Terminal 2: Frontend
make local-dev-frontend           # Hot Module Replacement (HMR)

# Option B: Background Start (Faster)
# ---------------------------------------------------
make local-dev-all                # Everything runs in background
make local-dev-status             # Check what's running

# ============================================================
# DEVELOPMENT CYCLE
# ============================================================
# 1. Make code changes in your editor
# 2. See changes instantly (auto-reload for both backend/frontend)
# 3. Test via http://localhost:3000
# 4. Run quality checks before commit
make quick-check

# ============================================================
# END OF DAY
# ============================================================
make local-dev-stop               # Stop all services

# First time ever
make local-dev-setup && make local-dev-all

# Every day after
make local-dev-all

🔧 Essential Commands Reference

🧪 Testing & Quality

# Atomic tests (fastest - no DB, no coverage)
make test-atomic

# Unit tests (mocked dependencies)
make test-unit-fast

# All local tests
make test-atomic && make test-unit-fast

# Ensure infrastructure is running
make local-dev-infra

# Run integration tests
make test-integration

# Full test suite
make test-all    # atomic + unit + integration

make quick-check       # Fast: format + lint (use before commit)
make lint              # Full linting (ruff + mypy)
make format            # Auto-fix formatting issues
make security-check    # Security scans (bandit + safety)
make coverage          # Generate coverage report (60% threshold)

🐳 Production & Container Workflows

# Build production images locally
make build-all

# Start production environment
make prod-start

# Check status
make prod-status

# View logs
make prod-logs

# Stop production
make prod-stop

# Pull latest images
docker compose -f docker-compose.production.yml pull

# Start production environment
make prod-start

📊 Features & Capabilities

🧠 Advanced AI Features

• Chain of Thought Reasoning: Automatic question decomposition with step-by-step reasoning, iterative context building, and transparent reasoning visualization
• Automatic Pipeline Resolution: Zero-config search experience - backend automatically selects and creates pipelines based on user context
• Token Tracking & Monitoring: Real-time usage tracking across all LLM interactions with detailed breakdowns
• Multi-LLM Support: Seamless switching between WatsonX, OpenAI, and Anthropic with provider-specific optimizations
• IBM Docling Integration: Enhanced document processing for complex formats (PDF, DOCX, XLSX)
• Question Suggestions: AI-generated relevant questions based on document collection content
• MCP Context Forge: Tool enrichment via IBM MCP with resilience patterns

🔍 Search & Retrieval

• Vector Databases: Pluggable support for Milvus (default), Elasticsearch, Pinecone, Weaviate, ChromaDB via common interface
• Hybrid Search: Combines semantic vector similarity with keyword search strategies
• Source Attribution: Granular document source tracking with chunk-level page references across reasoning steps
• Advanced Chunking: Hierarchical chunking strategies with configurable size and overlap
• Conversation History: Context-aware search with conversation memory for multi-turn interactions

🏗️ Architecture & Scalability

• Service-Based Design: 26+ services with clean separation of concerns and dependency injection pattern
• Repository Pattern: Data access abstraction layer for improved testability and maintainability
• Asynchronous Operations: Async/await throughout for efficient concurrent request handling
• Production Deployment: Docker + GHCR images, multi-stage builds, cloud-ready (AWS, Azure, GCP, IBM Cloud)
• Modular Design: Pluggable components for vector DBs, LLM providers, embedding models

🧪 Testing & Quality Assurance

• Comprehensive Test Suite: 947 automated tests across all layers (atomic, unit, integration, API, E2E)
• Multi-Layer Testing Strategy:
  • Atomic tests for schemas and data structures
  • Unit tests for business logic and services
  • Integration tests for service interactions
  • API tests for endpoint validation
• Security Scanning: Multi-layer security with Trivy (containers), Bandit (Python), Gitleaks + TruffleHog (secrets)
• Code Quality: Ruff linting, MyPy type checking, Pylint analysis, pre-commit hooks
• CI/CD Pipeline: Multi-stage GitHub Actions with test isolation, builds, and comprehensive integration testing

📚 Documentation

📖 Complete Documentation

• 📚 Full Documentation - Comprehensive guides and API reference
• 🚀 Getting Started - Quick start guide
• 🛠️ Development Guide - Development workflow and best practices
• 🧪 Testing Guide - Testing strategies and execution
• 🚀 Deployment Guide - Production deployment instructions

🔧 Configuration & Tools

• ⚙️ Configuration Guide - Environment setup and configuration
• 🔌 API Reference - Complete API documentation
• 🖥️ CLI Documentation - Command-line interface guide
• 🔐 Secret Management - Comprehensive guide for safe secret handling
• 🔗 MCP Integration - MCP Context Forge gateway setup and usage

🛠️ Command-Line Interface (CLI)

RAG Modulo includes a powerful CLI for interacting with the system:

# After installation, use the CLI commands:
rag-cli --help          # Main CLI help
rag-search              # Search operations
rag-admin               # Administrative tasks

# Example: Search a collection
rag-cli search query <collection-id> "your question here"

# Create a collection
rag-cli collection create --name "My Docs"

# Upload documents
rag-cli collection upload <collection-id> path/to/documents/

🚀 Deployment & Packaging

Production Deployment

RAG Modulo supports multiple deployment strategies:

1. Docker Compose (Recommended)

# Start production environment (all containers)
make prod-start

# Check status
make prod-status

# View logs
make prod-logs

# Stop production environment
make prod-stop

2. Pre-built Images from GHCR

# Pull latest images and start production environment
docker compose -f docker-compose.production.yml pull
make prod-start

Available Images:

• ghcr.io/manavgup/rag_modulo/backend:latest
• ghcr.io/manavgup/rag_modulo/frontend:latest

3. Custom Docker Deployment

# Build local images
make build-all

# Start production services
make prod-start

Cloud Deployment Options

Kubernetes Deployment

# Deploy with Helm chart
helm install rag-modulo deployment/helm/rag-modulo

🔄 CI/CD Pipeline

GitHub Actions Workflows

RAG Modulo uses a comprehensive CI/CD pipeline with multiple stages:

1. Code Quality & Testing (Separate Numbered Workflows)

On Every PR (parallel, ~2-3 min total):

• .github/workflows/01-lint.yml - Ruff, MyPy, Pylint, Pydocstyle (~60s)
• .github/workflows/04-pytest.yml - Unit tests with coverage (~90s)
• .github/workflows/07-frontend-lint.yml - ESLint for React/TypeScript (when frontend changes)

On Push to Main:

• .github/workflows/03-build-secure.yml - Docker builds + security scans
• .github/workflows/05-ci.yml - Integration tests (full stack)

2. Security Scanning (.github/workflows/02-security.yml)

Triggers: Push to main, Pull Requests

Secret Detection (3-Layer Defense):

1. Pre-commit hooks: detect-secrets with baseline (< 1 sec)
2. Local testing: Gitleaks via make pre-commit-run (~1-2 sec)
3. CI/CD: Gitleaks + TruffleHog (~45 sec)

Scans:

• Gitleaks: Pattern-based secret scanning with custom rules (.gitleaks.toml)
• TruffleHog: Entropy-based + verified secret detection
• Trivy: Container vulnerability scanning
• Bandit: Python security linting
• Safety: Python dependency vulnerabilities

⚠️ IMPORTANT: CI now fails on ANY secret detection (no continue-on-error). This ensures no secrets make their way to the repository.

Supported Secret Types:

• Cloud: AWS, Azure, GCP keys
• LLM: OpenAI, Anthropic, WatsonX, Gemini API keys
• Infrastructure: PostgreSQL, MinIO, MLFlow, JWT secrets
• Version Control: GitHub/GitLab tokens
• Generic: High-entropy strings, private keys

See: docs/development/secret-management.md for comprehensive guide

3. Documentation (.github/workflows/deploy-docs.yml)

Triggers: Push to main, changes to docs/

Actions:

• Build MkDocs site
• Deploy to GitHub Pages

Local CI Validation

Test CI pipeline locally before pushing:

# Run same checks as CI
make quick-check           # Lint and format validation
make test-unit-fast        # Unit tests

# Security checks
make security-check        # Bandit + safety scans
make pre-commit-run        # Full pre-commit hook suite

Pre-commit Hooks

Optimized for developer velocity:

On Commit (fast, 5-10 sec):

• Ruff formatting
• Trailing whitespace
• YAML syntax
• File size limits

On Push (slower, 30-60 sec):

• Atomic tests (schema validation)
• Fast unit tests (mocked dependencies)

In CI (comprehensive):

• All checks run regardless
• Ensures quality gates

Container Registry

GitHub Container Registry (GHCR):

• Automatic image builds on push
• Multi-architecture support (amd64, arm64)
• Image signing and verification
• Retention policies

Image Tags:

• latest: Latest main branch build
• sha-<commit>: Specific commit
• <branch>: Branch-specific builds
• v<version>: Release tags

🧪 Testing

Test Categories

Running Tests

# Fast local testing (no containers, recommended for development)
make test-unit-fast

# Specific test categories
make test-atomic       # Schema and data structure tests
make test-integration  # Service integration tests (requires infrastructure)
make test-e2e          # End-to-end tests

# Full test suite with coverage
make coverage

# Run specific test file
poetry run pytest tests/unit/services/test_search_service.py -v

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

Development Guidelines

1. Service Layer Architecture - Follow service-based patterns
2. Code Quality - Use type hints, comprehensive docstrings, PEP 8
3. Testing - Write tests for all new features
4. Documentation - Update docs for any changes

Contribution Process

1. Fork and Clone the repository
2. Create Feature Branch from main
3. Make Changes following our guidelines
4. Run Tests and ensure they pass
5. Submit Pull Request with clear description

📈 Roadmap

✅ Phase 1: Foundation (Completed)

• Service-based architecture with 26+ services
• Comprehensive test infrastructure (947 tests)
• Multi-LLM provider support (WatsonX, OpenAI, Anthropic)
• Vector database abstraction layer
• CI/CD pipeline with security scanning

✅ Phase 2: Advanced Features (Completed)

• Chain of Thought (CoT) reasoning system
• Automatic pipeline resolution
• Token tracking and monitoring
• IBM Docling integration
• Podcast generation with voice preview
• Question suggestion system
• Containerless local development workflow

🔄 Phase 3: Production Enhancement (Current)

• Production deployment with GHCR images
• Multi-stage Docker builds
• Security hardening (Trivy, Bandit, Gitleaks, TruffleHog)
• Enhanced monitoring and observability
• Performance optimization and caching
• Authentication system improvements (OIDC)

🚀 Phase 4: Enterprise Features (Next)

• Multi-tenant support
• Advanced analytics and dashboards
• Batch processing capabilities
• API rate limiting and quotas
• Advanced caching strategies

🔮 Phase 5: Innovation (Future)

• Multi-modal support (image, audio)
• Agentic AI workflows
• Real-time collaborative features
• Advanced reasoning strategies
• Federated learning support

🆘 Troubleshooting

Quick Diagnostic Commands

Run these first to identify issues:

make check-docker          # Verify Docker installation
make local-dev-status      # Check what's running
make venv                  # Ensure Python environment is set up
docker compose -f docker-compose-infra.yml ps  # Check infrastructure

Common Issues & Solutions

# Problem: "Command not found" errors
make venv                    # Creates Python venv (auto-installs Poetry)
source .venv/bin/activate

# Problem: Services won't start
make local-dev-stop          # Stop everything cleanly
make check-docker            # Verify Docker requirements
make local-dev-infra         # Restart infrastructure only

# Problem: Port conflicts
make local-dev-stop          # Stops all services and containers
lsof -i :8000 && kill $(lsof -t -i:8000)  # Kill backend on port 8000
lsof -i :3000 && kill $(lsof -t -i:3000)  # Kill frontend on port 3000

# Problem: Weird behavior after updates
make clean-venv              # Remove Python venv
make local-dev-setup         # Fresh install
make clean-all               # Nuclear option: clean everything

# Makefile auto-installs Poetry (recommended)
make venv

# OR install manually
curl -sSL https://install.python-poetry.org | python3 -

# Check Python version
python --version    # Should be 3.12+

# macOS: Install Python 3.12
brew install [email protected]

# Ubuntu/Debian: Install Python 3.12
sudo apt update
sudo apt install python3.12 python3.12-venv

# Method 1: Use Makefile (recommended)
make clean-venv
make local-dev-setup

# Method 2: Manual Poetry setup (from project root)
poetry config virtualenvs.in-project true
poetry cache clear . --all
poetry install --with dev,test --sync
source .venv/bin/activate

# Ensure you're in Poetry venv (from project root)
source .venv/bin/activate
which python    # Should show: .venv/bin/python
which ruff      # Should show: .venv/bin/ruff
ruff --version  # Should be 0.14.0+

# If wrong version, reinstall
poetry install --with dev,test --sync

# Verify PYTHONPATH and reinstall (from project root)
source .venv/bin/activate
poetry install --with dev,test --sync
python -c "import sys; print('\n'.join(sys.path))"

# Check if Docker is installed
docker --version
docker compose version

# If missing, install:
# macOS: brew install --cask docker
# Or download Docker Desktop: https://www.docker.com/products/docker-desktop

# Ubuntu/Debian
sudo apt update
sudo apt install docker-compose-plugin

# Verify
docker compose version

# Clean restart
make local-dev-stop
docker system prune -f
make local-dev-infra

# Check what failed
docker compose -f docker-compose-infra.yml ps
docker compose -f docker-compose-infra.yml logs

# Common fixes
docker volume prune -f        # Clear old volumes
make clean-all                # Nuclear option

# Find what's using ports
lsof -i :8000   # Backend
lsof -i :3000   # Frontend
lsof -i :5432   # PostgreSQL
lsof -i :19530  # Milvus

# Stop conflicting services
make local-dev-stop

# Kill specific port (if needed)
kill $(lsof -t -i:8000)

# Fix volume permissions
chmod -R 777 ./volumes
docker compose -f docker-compose-infra.yml down -v
make local-dev-infra

# Ensure venv is activated (from project root)
source .venv/bin/activate

# Install test dependencies
poetry install --with test --sync

# Run specific test with verbose output
poetry run pytest tests/unit/test_example.py -vv -s

# Ensure infrastructure is running
make local-dev-infra
make local-dev-status

# Wait for services to be ready
sleep 10

# Run tests
make test-integration

# Check PYTHONPATH (from project root)
source .venv/bin/activate
poetry run pytest tests/ --collect-only  # See what tests are collected

# If imports fail, reinstall
poetry install --with dev,test --sync

# Add to .env
SKIP_AUTH=true
DEVELOPMENT_MODE=true
ENABLE_MOCK_AUTH=true

# Restart backend
make local-dev-stop
make local-dev-backend

# Check backend logs
tail -f /tmp/rag-backend.log

# Restart backend with debug (from project root)
source .venv/bin/activate
LOG_LEVEL=DEBUG uvicorn main:app --reload --port 8000 --app-dir backend

# Clear cache and retry
cd frontend
rm -rf node_modules package-lock.json
npm cache clean --force
npm install

# Check Node version
node --version    # Should be 18+

# Update Node (macOS)
brew upgrade node

# Update Node (Linux - using nvm)
nvm install 18
nvm use 18

# Reinstall and start
cd frontend
rm -rf node_modules
npm install
npm run dev

# Clear build cache
cd frontend
rm -rf dist .vite
npm run dev

# Check if infrastructure is running locally
make local-dev-status

# Monitor resource usage
docker stats

# Check logs for bottlenecks
tail -f /tmp/rag-backend.log | grep -i "slow\|timeout\|error"

# Restart services to clear memory
make local-dev-stop
docker system prune -f
make local-dev-infra

🆘 Getting Help

Still Stuck?

1. Check service status: make local-dev-status
2. View logs: tail -f /tmp/rag-backend.log or make logs
3. Try clean restart: make local-dev-stop && make clean && make local-dev-all
4. Nuclear option: make clean-all (removes everything - be careful!)
5. Ask for help: Open an issue with:
   • Error message
   • Output of make local-dev-status
   • OS and Docker version
   • Steps to reproduce

📄 License

This project is licensed under the MIT License.

🙏 Acknowledgments

• IBM Docling - Advanced document processing and understanding
• IBM WatsonX - Enterprise AI foundation models
• FastAPI - Modern, high-performance web framework
• React - Powerful UI library for building interactive interfaces
• Milvus - High-performance vector database
• Docker - Containerization and deployment platform
• All Contributors - Thank you for your contributions!