MERaLiON Multilingual Audio LLM Chat Application

A web application for multilingual voice and text interactions with the MERaLiON-AudioLLM-Whisper-SEA-LION model. Supports English, Mandarin, Malay, and Tamil.

Architecture

┌─────────────────────────────────────────────┐
│  Frontend (Next.js + React)                 │
│  - Audio recording (MediaRecorder API)      │
│  - Text input                               │
│  - Language selection                       │
│  - Response display                         │
└──────────────┬──────────────────────────────┘
               │ HTTP/JSON
               ↓
┌─────────────────────────────────────────────┐
│  Backend (FastAPI)                          │
│  - Audio/text processing endpoints          │
│  - Model inference orchestration            │
│  - CORS support for frontend                │
└──────────────┬──────────────────────────────┘
               │ PyTorch/Transformers
               ↓
┌─────────────────────────────────────────────┐
│  AI Inference (GPU Server)                  │
│  - MERaLiON-AudioLLM model                  │
│  - Whisper audio processor                  │
│  - SEA-LION text decoder                    │
└─────────────────────────────────────────────┘

Setup Instructions

Prerequisites

• Python 3.9+
• Node.js 16+ (for frontend)
• NVIDIA GPU with 24GB+ VRAM (or use 4-bit quantization)
• Git

Backend Setup

1. Install Python dependencies:

cd backend
pip install -r requirements.txt

2. Get HuggingFace token:

   • Visit https://huggingface.co/settings/tokens
   • Create a new token with read access
   • Run huggingface-cli login and provide your token

3. Start the FastAPI server:

cd backend
python main.py

The API will be available at http://localhost:8000

Frontend Setup

1. Install dependencies:

cd frontend
npm install

2. Start the development server:

npm run dev

The app will be available at http://localhost:3000

Usage

1. Navigate to http://localhost:3000 in your browser
2. Select your preferred response language
3. Choose either:
   • Push to Talk: Click "Start Recording", speak, and submit
   • Text Input: Type your message and send

API Endpoints

POST /process-audio

Process audio and get text response

{
  "audio": <binary>,
  "language": "Singapore English" | "Mandarin" | "Malay" | "Tamil"
}

POST /process-text

Process text input and get response

{
  "text": "Your message here",
  "language": "Singapore English" | "Mandarin" | "Malay" | "Tamil"
}

GET /health

Health check endpoint

GET /languages

Get list of supported languages

Model Details

• Model ID: MERaLiON/MERaLiON-AudioLLM-Whisper-SEA-LION
• Audio Processor: OpenAI Whisper
• Quantization: 4-bit (BitsAndBytes) for reduced VRAM usage
• Max Tokens: 256 per response

Supported Languages

• Singapore English
• Mandarin Chinese
• Malay
• Tamil

Hardware Requirements

Minimum

• 1x NVIDIA GPU with 24GB VRAM (RTX 3090, RTX 4090, A100)
• OR CPU with 32GB+ RAM (very slow)

Recommended

• NVIDIA A100 (40GB)
• NVIDIA H100 (80GB)

Development

File Structure

├── backend/
│   ├── main.py              # FastAPI server
│   └── requirements.txt      # Python dependencies
├── frontend/
│   ├── pages/               # Next.js pages
│   ├── components/          # React components
│   ├── styles/              # CSS files
│   ├── package.json         # Node dependencies
│   └── next.config.js       # Next.js config
└── README.md

Troubleshooting

Model loading takes too long

• This is normal on first load (downloading 10+ GB)
• Subsequent runs use cached model

CUDA out of memory

• Increase max_new_tokens in backend/main.py to lower value
• Consider using a smaller model or more powerful GPU

Microphone not working

• Check browser permissions (chrome://settings/content/microphone)
• Ensure HTTPS or localhost is used (browsers restrict microphone on HTTP)

CORS errors

• Verify frontend is running on port 3000
• Check API_BASE_URL in frontend/.env.local

License

This project uses the MERaLiON model from SEA-LION. See model card for details.