Voice-enabled RAG Chatbot

A end-to-end conversational system that accepts voice input, transcribes it, retrieves relevant context from a vector database, and generates grounded answers using a large language model.

Overview

This project implements a Retrieval-Augmented Generation (RAG) pipeline with voice as the primary input modality. The system is designed to answer questions about specific topics (e.g., Wikipedia articles) by combining:

• Speech-to-Text (ASR): Converts voice input to text using Indic-Conformer
• Language Translation: Translates non-English text to English using Sarvam API
• Vector Retrieval: Searches a Chroma vector database for relevant context
• Generative AI: Uses LLaMA-3.1-8B via GitHub Models for answer generation
• User Interface: Streamlit web application for easy interaction

Architecture

┌─────────────┐
│  Audio File │
└──────┬──────┘
      │
      ▼
┌──────────────────┐
│  ASR (FastAPI)   │
│  Indic-Conformer │
└──────┬───────────┘
      │
      ▼
┌──────────────────────┐
│  Translation Layer   │
│  (Sarvam API)        │
└──────┬───────────────┘
      │
      ▼
┌──────────────────────┐
│  Vector Retrieval    │
│  (Chroma DB)         │
└──────┬───────────────┘
      │
      ▼
┌──────────────────────┐
│  LLM Query Engine    │
│  (GitHub Models)     │
└──────┬───────────────┘
      │
      ▼
┌──────────────────────┐
│  Formatted Answer    │
│  (Streamlit UI)      │
└──────────────────────┘

Features

• Multilingual Support: Accepts queries in Indian languages (Hindi, Tamil, Telugu, etc.)
• Context-Aware Responses: Uses semantic search to retrieve relevant document chunks
• Low-Latency Pipeline: Optimized inference with proper error handling
• Scalable Architecture: Modular design allows easy replacement of components
• User-Friendly Interface: Interactive Streamlit application with chat history

Prerequisites

System Requirements

• Python 3.9+
• 8GB RAM (recommended)
• CUDA-capable GPU (optional, for faster inference)

API Keys Required

1. GITHUB_TOKEN: For GitHub Models API access (sign up here)
2. SARVAM_API_KEY: For translation service (get free credits)

Installation

Step 1: Clone the Repository

git clone <your-repo-url>
cd voice-rag-chatbot

Step 2: Create Virtual Environment

python -m venv venv

# On Windows
venv\Scripts\activate

# On macOS/Linux
source venv/bin/activate

Step 3: Install Dependencies

pip install -r requirements.txt

Step 4: Configure Environment Variables

Copy .env.example to .env and fill in your API keys:

cp .env.example .env

Edit .env:

GITHUB_TOKEN=your_github_models_token
SARVAM_API_KEY=your_sarvam_api_key
ASR_ENDPOINT=http://localhost:8000/transcribe
CHROMA_DB_PATH=./chroma_db
COLLECTION_NAME=wiki_article

Usage

Step 1: Collect and Prepare Data

Fetch a Wikipedia article for your knowledge base:

python data.py "Apple fruit"

This will:

• Search for the closest Wikipedia article matching your query
• Extract and clean the article text
• Save it as Apple_fruit.txt

Supported Queries: Any topic with a Wikipedia article (e.g., "Machine Learning", "Climate Change", "Ancient Rome")

Other Approaches for WebScraping: BeautifulSoup

Step 2: Create Vector Database

Transform the article text into embeddings and build a searchable index:

python vectordb.py Apple_fruit.txt --db-path ./chroma_db --chunk-size 1000

Key Parameters:

• --chunk-size: Size of text chunks (1000 chars recommended for semantic coherence, especially for Wikipedia articles)
• --chunk-overlap: Overlap between chunks (200 chars recommended to preserve context between chunks)
• --model-name: Embedding model (default: sentence-transformers/all-MiniLM-L6-v2)

Why these settings?

• Chunk Size (1000 chars): Large enough to preserve semantic meaning while small enough for precise retrieval
• Overlap (200 chars): Prevents losing context at chunk boundaries; roughly 20% overlap
• Embedding Model: Lightweight yet effective for semantic search; fast inference with reasonable accuracy

Step 3: Deploy ASR Endpoint

Start the FastAPI server for audio transcription:

uvicorn asr:app --host 0.0.0.0 --port 8000

Endpoint: POST http://localhost:8000/transcribe

• Input: Audio file (WAV, MP3, FLAC, M4A)
• Output: JSON with transcribed text and detected language

Supported Languages: Hindi, Tamil, Telugu, Kannada, Malayalam, and more (see Indic-Conformer model)

Step 4: Run the Chatbot UI

Launch the Streamlit application:

streamlit run app.py

The UI will open at http://localhost:8501

Usage:

1. Upload an audio file (WAV, MP3, FLAC, M4A)
2. Click "Ask"
3. View the transcribed question and AI-generated answer
4. Browse chat history in the interface

Project Structure

voice-rag-chatbot/
├── app.py                 # Streamlit UI application
├── asr.py                 # FastAPI ASR endpoint
├── data.py                # Wikipedia article scraping
├── vectordb.py            # Vector database creation
├── txt_translate.py       # Sarvam translation API wrapper
├── voice_rag_chatbot.py   # Main RAG pipeline
├── requirements.txt       # Python dependencies
├── .env.example           # Environment template
├── .gitignore            # Git exclusions
└── README.md             # This file

Design Decisions & Observations

1. Vector Database Choice (Chroma)

Why Chroma?

• Lightweight and easy to set up (no external server required)
• Excellent LangChain integration
• Persistent storage with SQLite backend
• Flexible metadata support

Advantages:

• Fast similarity search using FAISS or Annoy backends
• Simple Python API for development
• Good for prototyping and small-to-medium datasets

Drawbacks:

• Not ideal for massive datasets (>10M documents)
• Limited to single-node deployment
• No built-in distributed querying

Alternative Considerations: For production scale, consider Pinecone, Weaviate, or Milvus.

2. Embedding Model Selection (all-MiniLM-L6-v2)

Why this model?

• Produces 384-dimensional embeddings (vs 1536 for OpenAI)
• 33M parameters (small, fast inference)
• High semantic quality for its size
• Freely available on Hugging Face Hub

Trade-offs:

• Smaller than large models like E5-Large, but sufficient for medium-length documents
• ~50ms inference per text chunk on CPU

3. ASR Model (Indic-Conformer)

Why this model?

• Specifically trained for Indian languages
• Conformer architecture (combines CNNs + Transformers)
• 600M parameters provides good accuracy-to-speed trade-off
• Support for multiple languages in one model

Performance Notes:

• First inference includes model loading (~2-3 seconds)
• Subsequent requests: ~0.5-1 second for 5-second audio
• Consider caching the model in production

4. LLM Choice (LLaMA-3.1-8B)

Why GitHub Models + LLaMA-3.1-8B?

• Free API tier with reasonable quotas
• Good balance of quality and latency
• Supports RAG-style prompting
• No local GPU required

Alternative: Can easily swap for other models (Mistral, Orca, etc.) by changing GITHUB_MODEL env var

5. Translation Service (Sarvam API)

Why not translate locally?

• Local translation models are lower quality for Indian languages
• Sarvam API specifically optimized for Indian language pairs
• Free 1000 credits on signup (sufficient for testing)

Error Handling & Edge Cases

The system handles:

1. Missing API Keys: Gracefully fails with informative error messages
2. Network Issues: Retry logic for ASR and API calls (default: 60s timeout)
3. Empty/Invalid Audio: Caught and reported to user
4. Non-existent Files: Validated before processing
5. Vector DB Not Found: Creates new DB or loads existing
6. Model Loading Failures: Wrapped in try-except with fallback

Performance Considerations

Optimization Tips:

• Use GPU for ASR and embeddings (100x+ speedup)
• Cache embeddings model in memory
• Reduce max_tokens in LLM if latency is critical
• Use batch processing for multiple queries

Challenges & Solutions

Challenge 1: ASR for Non-Standard Audio

Problem: Audio quality varies; background noise affects transcription. Solution:

• Preprocess audio (resample to 16kHz, normalize amplitude)
• Implement audio validation before ASR
• Consider noise reduction filters for production

Challenge 2: Hallucinations in LLM Responses

Problem: LLM may generate plausible-sounding but incorrect answers. Solution:

• Low temperature (0.2) to reduce randomness
• Require context chunks for every answer
• Add confidence scores or source citations
• Validate answers against retrieved context

Challenge 3: Language Detection Accuracy

Problem: User might ask in mixed language or code-switching. Solution:

• Sarvam API auto-detects source language
• Pipeline handles any detected language
• Consider explicit language hints if needed

Challenge 4: Stale or Incomplete Vector DB

Problem: Forgetting to regenerate DB after data updates. Solution:

• Document DB creation step in setup
• Add version tracking or timestamps
• Implement incremental updates for large datasets

Testing

Manual Testing

1. Test ASR alone:

   curl -X POST http://localhost:8000/transcribe \
     -F "file=@test_audio.wav"
   

2. Test Translation:

   python txt_translate.py "नमस्ते कैसे हो"
   

3. Test Vector DB:

   from vectordb import retrieve_chunks
   chunks = retrieve_chunks("apple farming")
   print(chunks)
   

4. Test Full Pipeline:

   from voice_rag_chatbot import voice_rag_pipeline
   answer = voice_rag_pipeline("path_to_audio.wav")
   print(answer)
   

Deployment Recommendations

Local Development

Use the setup above—perfectly suitable for prototyping.

Staging/Production

1. Containerize with Docker:

   FROM python:3.10
   WORKDIR /app
   COPY requirements.txt .
   RUN pip install -r requirements.txt
   COPY . .
   CMD ["streamlit", "run", "app.py"]
   

2. Deploy ASR separately (GPU-accelerated VM)

3. Use managed vector DB (Pinecone, Weaviate cloud)

4. Scale LLM calls with request queuing

5. Add monitoring (Sentry, CloudWatch, Prometheus)

Future Enhancements

• Support for document upload (PDF, DOCX)
• Speaker identification and multi-turn conversations
• Fine-tuned embeddings for domain-specific queries
• Caching frequently asked questions (Q&A pairs)
• A/B testing different LLM models
• Source citation and answer confidence scores
• Batch processing for multiple documents
• Multi-language output (answer in query language)

Troubleshooting

Issue: "ASR endpoint unreachable"

Solution: Ensure ASR server is running: uvicorn asr:app --port 8000

Issue: "GITHUB_TOKEN not set"

Solution: Add token to .env file and restart Streamlit app

Issue: "Vector DB not found"

Solution: Run python vectordb.py your_file.txt first

Issue: "Slow responses"

Solution:

• Check GPU availability: torch.cuda.is_available()
• Reduce model chunk_size for faster retrieval
• Use smaller embedding model

AI Declaration

The work uses Claude Opus model for generating documentation assistance and code fixes. All generated content has been reviewed and edited for accuracy.

License

This project is open-sourced under the MIT License. See LICENSE file for details.

Contributing

Contributions welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Submit a pull request with clear descriptions

References

• Indic-Conformer Model
• Sarvam API Documentation
• GitHub Models API
• LangChain Documentation
• Chroma Vector DB
• Streamlit Documentation

Contact & Support

For issues, questions, or suggestions:

• Open an issue on GitHub
• Check existing documentation
• Review error logs in console output

---

Last Updated: January 2026
Python Version: 3.9+
Status: Active Development