SyncVSR for Gujarati Visual Speech Recognition

First attempt at building a Visual Speech Recognition (VSR) system for Gujarati language using the SyncVSR framework.

Project Overview

This project implements the SyncVSR (Synchronize Visual Speech Recognition) framework for Gujarati — a language with no existing lip reading datasets or benchmarks.

Key Contributions

1. Gujarati VSR Pipeline: End-to-end pipeline from YouTube videos → lip crops → AV-HuBERT features → Conformer training
2. SyncVSR Adaptation: Adapted the SyncVSR framework (originally for English) to Gujarati using MMS-300m for audio tokenization
3. Gu-LRW Dataset (in progress): Building the first word-level lip reading dataset for Gujarati (30 word classes)
4. Ablation Study: Systematic comparison of baseline, Conformer, and Conformer+SyncVSR architectures

Architecture

Video Frames → AV-HuBERT (frozen) → Conformer Encoder → Classification Head → Word
                                          ↓
                                    Sync Head → Audio Token Prediction (training only)

Loss = L_classification + λ × L_sync

Project Structure

├── gujarati_vsr/                    # Core code
│   ├── lightning_phase2_train.py    # Sentence-level SyncVSR training
│   ├── word_level_syncvsr.py       # Word-level experiments + ablation
│   └── ...                         # Supporting scripts
│
├── docs/                            # Documentation
│   ├── dataset_protocol.md          # Gu-LRW recording protocol
│   └── syncvsr_explained.md         # Paper walkthrough
│
├── SyncVSR/                         # Original SyncVSR repo (reference)
└── auto_avsr/                       # Auto-AVSR repo (reference)

Dataset

Sentence-Level (5K clips)

• Source: YouTube Gujarati speech videos
• 5,216 clips with transcriptions
• Used for initial sentence-level experiments

Word-Level: Gu-LRW (in progress)

• 30 Gujarati word classes
• 5+ speakers, 15+ takes per word
• First word-level lip reading dataset for any Indian language

Training Results

Sentence-Level (5K samples → insufficient data)

Finding: 5K samples insufficient for sentence-level VSR. Model learns language patterns but not lip-to-text mapping.

Word-Level Ablation (pending Gu-LRW dataset)

Requirements

• Python 3.8+
• PyTorch 2.0+
• torchaudio
• transformers (HuggingFace)
• sentencepiece
• tqdm

Pretrained Models Used

References

• SyncVSR Paper (Ahn et al., 2024)
• AV-HuBERT
• MMS: Scaling Speech Technology to 1000+ Languages

Course

IT549 — Advanced Machine Learning, Course Project