CIU Chatbot - RAG System v2.2

Production-ready Retrieval-Augmented Generation system for Cyprus International University

CIU Chatbot is an intelligent RAG system that answers questions from CIU staff and students using advanced semantic search and language generation.

Key Features

| Multilingual Support | English, Turkish, French (auto-detection) | | Hybrid Retrieval | Dense embeddings + BM25 lexical search + Cross-Encoder reranking | | FAISS Indexing | 2.2M+ documents, ultra-fast vector search | | HTML Cleaning | Automatic tag removal and whitespace normalization | | Semantic Chunking | 3 adaptive strategies: generic, FAQ, hierarchical | | FastAPI Server | GPU/CPU optimized, production-ready | | Comprehensive Logging | JSON logs, file rotation, 7-day retention |

Data Support

• Excel: 132,221 ticket documents (perfect_ticket_data.xlsx)
• PDF: 63 documents from 3 files
• Validation: 95.3% valid documents after cleaning
• Embeddings: Sentence-Transformers (all-mpnet-base-v2, 768D)

Quick Start

Installation (1 minute)

# Clone and navigate
cd /home/user/ciu-chatbot

# Create Python environment
python3 -m venv venv
source venv/bin/activate        # Linux/Mac
# venv\Scripts\activate          # Windows

# Install dependencies
pip install -r requirements.txt
pip install openpyxl            # Optional: for Excel support

Configuration (1 minute)

Create .env file in project root:

# API Settings
API_HOST=0.0.0.0
API_PORT=8000
API_WORKERS=4

# Embedding Model
EMBEDDING_MODEL_NAME=sentence-transformers/all-mpnet-base-v2
EMBEDDING_DIM=768

# RAG Parameters
CHUNK_SIZE=500
CHUNK_OVERLAP=100
RETRIEVAL_TOP_K=10
RERANKING_TOP_K=5
MIN_SIMILARITY_THRESHOLD=0.3

# Chunker Type (generic | faq | hierarchical)
CHUNKER_TYPE=generic

# Performance
USE_GPU=true
LOG_LEVEL=INFO
DEBUG=false

Build Index (3 minutes)

python3 build_optimized_rag.py \
  data/raw/perfect_ticket_data.xlsx \
  data/raw/pdfs/ \
  --output data/complete_index \
  --rebuild

Output:

Total documents: 132,284
Valid documents: 126,033 (95.3%)
Generated chunks: 180,500
Embedded vectors: 180,500
FAISS index built successfully (~150MB)

Start System (1 minute)

# Option A: Automatic (Linux/Mac)
./START.sh

# Option B: Manual
uv run python -m src.api.fastapi_app_v2

# Option C: Docker
docker build -t ciu-chatbot:2.2 .
docker run -p 8000:8000 ciu-chatbot:2.2

Server Running:

INFO:     Uvicorn running on http://0.0.0.0:8000
INFO:     Application startup complete

5️⃣ Test API (1 minute)

# Interactive docs
curl http://localhost:8000/docs

# Test search
curl -X POST http://localhost:8000/search \
  -H "Content-Type: application/json" \
  -d '{
    "query": "How to register courses?",
    "top_k": 5,
    "language": "en"
  }'

# Health check
curl http://localhost:8000/health

---

📖 API Documentation

Endpoints

POST /search - Search with retrieval

Request:

{
  "query": "How to register?",
  "top_k": 5,
  "language": "en",
  "use_reranking": true,
  "confidence_threshold": 0.3
}

Response:

{
  "query": "How to register?",
  "language_detected": "en",
  "results": [
    {
      "text": "To register for courses...",
      "score": 0.92,
      "chunk_index": 145,
      "metadata": {
        "ticket_number": "TICKET-998",
        "source": "Support System"
      }
    }
  ],
  "confidence": 0.89,
  "timing": {"total_ms": 234}
}

GET /health - System status

GET /system-info - System capabilities

POST /search-contextual - Search with context

---

🏗️ Project Structure

ciu-chatbot/
├── src/
│   ├── api/              # FastAPI endpoints
│   ├── data/             # Data processing & chunking
│   ├── models/           # Embeddings & LLM
│   ├── retrieval/        # Search & reranking
│   ├── agents/           # Agentic RAG
│   ├── utils/            # Utilities & helpers
│   └── processing/       # Batch processing
│
├── data/
│   ├── raw/              # Source data (Excel, PDFs)
│   ├── complete_index/   # FAISS index + chunks
│   └── processed/        # Processed data
│
├── config/
│   ├── settings.py       # Configuration
│   └── logging_config.py # Logging setup
│
├── tests/                # Test suite
├── build_optimized_rag.py # Index builder
├── requirements.txt      # Dependencies
└── README.md

💻 Usage Examples

Python API

from src.retrieval.advanced_retriever import AdvancedRetriever
from src.models.embedding import EmbeddingModel
from config.settings import settings

# Initialize
embedding_model = EmbeddingModel(settings.EMBEDDING_MODEL_NAME)
retriever = AdvancedRetriever(
    embedding_model=embedding_model,
    index_path=settings.FAISS_INDEX_PATH
)

# Search
results = retriever.retrieve(
    query="How to register?",
    k=5,
    threshold=0.3
)

# Display results
for result in results:
    print(f"Score: {result['score']:.2f} | {result['text'][:100]}...")

Batch Processing

from src.processing.batch_processor import BatchProcessor

processor = BatchProcessor()
results = processor.process_queries([
    "How to register courses?",
    "What are admission requirements?",
    "How to get a transcript?"
])

---

System Architecture

User Query
    ↓
[Language Detection] ────────────→ Auto-detect (EN/TR/FR)
    ↓
[Query Embedding] ────────────────→ Sentence-Transformers
    ↓
[Dense Retrieval] ────────────────→ FAISS search (top k×2)
    ↓
[Hybrid Scoring] ─────────────────→ Semantic + BM25 lexical
    ↓
[Reranking] (Optional) ───────────→ Cross-Encoder precision
    ↓
[Top-K Results] ──────────────────→ With metadata
    ↓
[LLM Generation] (Optional) ──────→ Ollama or other LLM
    ↓
[Response Formatting] ────────────→ Structured JSON
    ↓
Response to User

---

Core Features

1️⃣ Semantic Chunking

# 3 adaptive strategies:

# Generic - for free text (default)
- Respects sentence boundaries
- Flexible chunk size with overlap
- Perfect for support tickets

# FAQ-Specific - for Q&A pairs  
- Keeps question+answer together
- Preserves document structure
- Enriched metadata

# Hierarchical - for documents
- Respects document hierarchy
- Identifies sections/chapters
- Optimal context preservation

2️⃣ Automatic HTML Cleaning

Removes: tags, scripts, styles, entities, control characters Result: Clean, readable text

3️⃣ Hybrid Retrieval Pipeline

• Dense Search: FAISS with L2 distance
• Lexical Scoring: BM25-like ranking
• Reranking: Cross-Encoder refinement
• Confidence Filter: Adjustable threshold

4️⃣ Multilingual Support

Auto-detects and processes: English, Turkish, French

Production Optimized

---

🧪 Testing & Validation

Sample Queries

queries = {
    "en": [
        "How to register courses?",
        "What are admission requirements?",
        "How to get a transcript?",
    ],
    "tr": [
        "Ders kaydı nasıl yapılıyor?",
        "Kabul şartları nelerdir?",
        "Transkript nasıl alınır?",
    ]
}

Index Statistics

python3 -c "
from src.retrieval.faiss_retriever import FAISSRetriever
from config.settings import settings

retriever = FAISSRetriever(settings.FAISS_INDEX_PATH, settings.CHUNKS_PATH)
stats = retriever.get_stats()
print('Index Stats:')
for key, value in stats.items():
    print(f'  {key}: {value}')
"

Validation Report

# Run test suite
uv run pytest tests/ -v

# Coverage report
uv run pytest tests/ --cov=src --cov-report=html

---

🐛 Troubleshooting

Common Commands

# View recent logs
tail -f logs/app.log

# Profile system
uv run python -m cProfile -s cumtime -m src.api.fastapi_app_v2

# Memory profiling
uv pip install memory-profiler
uv run python -m memory_profiler src/api/fastapi_app_v2.py

# Clear cache
rm -rf __pycache__ .pytest_cache .egg-info dist build .venv

---

� Performance Metrics

Accuracy

Latency

Scalability

• Chunks: 180,500+
• Languages: 3 (EN/TR/FR)
• Concurrent Requests: 4-8 per worker
• Memory Footprint: ~200MB (FAISS + Model)

---

🔗 API Reference

Request/Response Format

/search - Semantic Search

Metadata Fields

{
  "text": "Chunk text",
  "score": 0.92,
  "chunk_index": 145,
  "metadata": {
    "ticket_number": "TICKET-998",
    "source": "Support System",
    "date": "2025-01-01"
  }
}

---

📥 Data Format

Input: Excel File

Required columns:

Input: FAQ CSV (Optional)

---

� Docker Deployment

Build Image

FROM python:3.11-slim

WORKDIR /app

# Install system dependencies
RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/*

# Setup Python
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy application
COPY . .
RUN mkdir -p logs data/faiss_index

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \
  CMD curl -f http://localhost:8000/health || exit 1

# Expose & run
EXPOSE 8000
CMD ["python3", "-m", "src.api.fastapi_app_v2"]

Run Container

# Build
docker build -t ciu-chatbot:2.2 .

# Run
docker run -p 8000:8000 \
  -v $(pwd)/data:/app/data \
  -v $(pwd)/logs:/app/logs \
  --env OLLAMA_HOST=http://host.docker.internal:11434 \
  ciu-chatbot:2.2

# Or with docker-compose
docker-compose up -d

Production with Gunicorn

gunicorn \
  --workers 4 \
  --worker-class uvicorn.workers.UvicornWorker \
  --bind 0.0.0.0:8000 \
  --timeout 60 \
  src.api.fastapi_app_v2:app

---

✅ Production Checklist

• Semantic chunking (3 strategies)
• HTML cleaning & normalization
• FAISS vector indexing
• Hybrid retrieval pipeline
• Cross-Encoder reranking
• Multilingual support (EN/TR/FR)
• FastAPI production server
• Error handling & validation
• Comprehensive logging
• Performance optimized
• Docker ready
• Full documentation

Status: 🟢 PRODUCTION READY v2.2

---

� Documentation

---

💡 Support

Issues & Debugging

1. Check logs: tail -f logs/app.log
2. View config: cat .env | grep -v "^#"
3. Test health: curl http://localhost:8000/health
4. Inspect index: ls -lh data/complete_index/

Contributing

# Setup development with uv
uv venv
source .venv/bin/activate
uv pip install -r requirements.txt
uv pip install pytest pytest-cov black flake8

# Run tests
uv run pytest tests/ -v

# Format code
uv run black src/ tests/

# Lint
uv run flake8 src/ tests/

Performance Profiling

# CPU profiling
uv run python -m cProfile -s cumtime -m src.api.fastapi_app_v2

# Memory profiling
uv pip install memory-profiler
uv run python -m memory_profiler src/api/fastapi_app_v2.py

# Load testing
uv pip install locust
uv run locust -f tests/locustfile.py

---

📝 Changelog

v2.3 (February 2026)

• 🚀 Integrated uv package manager for faster installs
• 📊 Updated with actual project metrics (2.2M documents, 348MB index)
• 🗂️ Clarified index structure (cleaned_documents.json)
• 📈 Accurate data sizes and memory footprint
• 🔧 Updated all commands to use uv
• 🐧 Added START.sh for Linux/Mac startup

v2.2 (January 2026)

• 📖 Improved README with better formatting
• 📊 Enhanced tables and organization
• 🎯 Clearer quick start guide
• 🔧 Better troubleshooting section
• 📈 Updated performance metrics

v2.1 (November 2025)

• ✨ Automatic HTML cleaning
• 🧹 Chunk validation on load
• 📚 Improved documentation
• 🐛 Import fixes
• ⚡ Performance optimization

v2.0 (October 2025)

• 🎯 Semantic chunking system
• 🔄 Hybrid retrieval pipeline
• 🌍 Multilingual support
• 📊 FAISS indexing
• 🚀 FastAPI implementation

---

📄 License

© 2025 Cyprus International University - All Rights Reserved

---

Last Updated: February 20, 2026
Version: 2.3
Status: 🟢 Production Ready

---

Quick Links

• 🔗 API Documentation - Interactive Swagger UI
• 📊 System Info - Runtime metrics
• 🐛 Health Check - System status
• 📁 Data Directory - Index and source files
• 📝 Logs Directory - Application logs