Vistaar: Diverse Benchmarks and Training sets for Indian Language ASR

Vistaar is a set of 59 benchmarks and training datasets across various language and domain combinations such as news, education, literature, tourism etc. The training datasets are avaialable for 12 Indian languages amounting to over 10,700 hours of labelled audio data. We also train IndicWhisper models by fine-tuning the Whisper models on the Vistaar train dataset and observe that it has the lowest WER on 39 out of 59 Vistaar benchmarks.

Benchmarks

Vistaar consists of benchmarks from several public datasets - Kathbath, FLEURS, CommonVoice, IndicTTS, MUCS, GramVaani across 12 languages. We evaluate IndicWhisper on these benchmarks along with 3 publicly available ASR systems and 2 commercially available systems. Below mentioned are the results

Word error rates (%) of IndicWhisper on the Vistaar Benchmark. Values in bold indicates benchmarks where IndicWhisper has the lowest WER.

Comparison of publicly-available models on the Hindi subset of the Vistaar benchmark

Table of contents

• Vistaar
  • Benchmarks
  • Table of contents
  • Resources
    • Download Training Datasets and Benchmarks
    • Download Models
  • Tutorials
    • Setting up your environment
    • Evaluating ASR models
      • Manifest creation
      • Running evaluation
    • Inference using IndicWhisper
      • Sample structure of manifest file
      • Running inference
    • Training on Vistaar Train Datasets
      • Manifest creation
      • Running training
  • Cite
  • License
  • Contributors
  • Contact

Resources

Download Training Datasets and Benchmarks

Download Models

Tutorials

Setting up your environment

• Setting up python virtual environment

  python -m venv <env_name>
  source <env_name>/bin/activate
  

• Installing/Updating Libraries

  sudo apt install ffmpeg
  pip install -r requirements.txt
  

Evaluating ASR models

• Manifest creation
  • For each dataset Download and extract the benchmark data in a directory. The data should be extracted in such a way that each folder inside should contain data for a particular language i.e each language specific folder should contain train, valid and test folder and within them the audio + transcript.txt
  • Sample structure of folder tree:

    kathbath
        ├── bengali
        │   ├── audio
        │   └── transcript.txt
        │
        └── gujarti
            ├── audio
            └── transcript.txt 
        .
        .
        .
        .
  

  • The manifest needs to be a Json file where each line is a dictionary with audio filepath, duration of audio and text transcript

   {"audio_filepath": <path to audio file 1>, "duration": <duration of audio file 1>, "text": <transcript of audio file 1>}
   {"audio_filepath": <path to audio file 2>, "duration": <duration of audio file 2>, "text": <transcript of audio file 2>}
   .
   .
   .
  

• Running evaluation

  python evaluation.py --model_path=<model path> \
  --manifest_path=<manifest path in vistaar> \
  --manifest_name=<dataset name in vistaar> \
  --device=<gpu to use> \
  --batch_size=<batch size> \
  --language=<2 letter language code>
  

Inference using IndicWhisper

• Sample structure of manifest file

{"audio_filepath":<path to audio file 1>}
{"audio_filepath":<path to audio file 2>}
.
.
.

• Running batch inference

deepspeed --include localhost:<gpus to include> \
transcribe.py <manifest path> \
<model path> \
<current language> \
<batch size>
<output path>

• Running inference for a single audio file

from transformers import pipeline

model_path = "hindi_models/whisper-medium-hi_alldata_multigpu"
device = "cuda"
lang_code = "hi"

whisper_asr = pipeline(
    "automatic-speech-recognition", model=model_path, device=device,
)

# Special case to handle odia since odia is not supported by whisper model
if lang_code == 'or':
    whisper_asr.model.config.forced_decoder_ids = (
        whisper_asr.tokenizer.get_decoder_prompt_ids(
            language=None, task="transcribe"
        )
    )
else:
    whisper_asr.model.config.forced_decoder_ids = (
        whisper_asr.tokenizer.get_decoder_prompt_ids(
            language=lang_code, task="transcribe"
        )
    )

result = whisper_asr("audio.mp3")
print(result["text"])

Training on Vistaar Train Datasets

• Manifest creation
  • Follow the steps as in Evaluating ASR models for the vistaar training datasets
• Running training

deepspeed --include localhost:<gpus to include> training.py \
--deepspeed=<path to deepspeed config file> \
--model_name_or_path=<model path> \
--dataset_name=<dataset language directory path> \
--language=<language> \
--train_split_name=<train manifest name> \
--eval_split_name=<validation manifest name> \
--max_steps="5000" \
--output_dir=<output directory path> \
--cache_dir=<cache directory for downloaded models> \
--per_device_train_batch_size="64" \
--per_device_eval_batch_size="32" \
--gradient_accumulation_steps="1" \
--logging_steps="10" \
--learning_rate="1e-5" \
--warmup_steps="500" \
--evaluation_strategy="steps" \
--eval_steps="500" \
--save_strategy="steps" \
--save_steps="500" \
--generation_max_length="225" \
--length_column_name="input_length" \
--max_duration_in_seconds="30" \
--text_column_name="sentence" \
--freeze_feature_encoder="False" \
--report_to="tensorboard" \
--metric_for_best_model="wer" \
--greater_is_better="False" \
--load_best_model_at_end \
--gradient_checkpointing \
--fp16 \
--do_train \
--do_eval \
--predict_with_generate \
--do_normalize_eval="False" \
--streaming="True" \
--use_auth_token="True" \
--push_to_hub="True"

License

Vistaar is MIT-licensed. The license applies to all the fine-tuned language models

Contributors

• Kaushal Bhogale, (AI4Bharat)
• Sai Narayan Sundaresan, (IITKGP, AI4Bharat)
• Abhigyan Raman, (AI4Bharat)
• Tahir Javed, (IITM, AI4Bharat)
• Mitesh Khapra, (IITM, AI4Bharat, RBCDSAI)
• Pratyush Kumar, (Microsoft, AI4Bharat)

Contact

• Mitesh Khapra (miteshk@cse.iitm.ac.in)
• Pratyush Kumar (pratyush@cse.iitm.ac.in)

Acknowledgements

We would like to thank EkStep Foundation for their generous grant which helped in setting up the Centre for AI4Bharat at IIT Madras to support our students, research staff, data and computational requirements. We would like to thank The Ministry of Electronics and Information Technology (NLTM) for its grant to support the creation of datasets and models for Indian languages under its ambitions Bhashini project. We would also like to thank the Centre for Development of Advanced Computing, India (C-DAC) for providing access to the Param Siddhi supercomputer for training our models. Lastly, we would like to thank Microsoft for its grant to create datasets, tools and resources for Indian languages.