Web app for transcribing audio files of LiveATC

This is a web app that transcribes audio files of LiveATC. We have three version of transcribe context, by Whisper, fine-tuned by ATCO2, and Whisper by promot.

How we built it

1. Preparation

1. Audio Collection & Segmentation

   • Randomly divide the 2-hours recordings into three equal parts (A, B, C).

   • Ensure a mix of difficulty levels in each part.

     • 2 hours of recordings -> 120 minutes.
     • Randomly select 40 minutes from each part, and assign them to A, B, and C.
     • Assume there are three students to help us transcribe the audio.

     Student	Stu1	Stu2	Stu3
     Reference 1	A	B	C
     Reference 2	B	C	A
     Reference 3	C	A	B

   • Requirements of transcriptions:

     • For better readability, all digits pronunciation will be written in digits, such as zero -> 0.
     • For the decimal(.) and comma (,), we will use the word "decimal" and "comma" respectively.
     • For the alphabet, we will use the NATO phonetic alphabet, we keep the original spelling.
     • If the speaker speeks in a different language, we will write the word in the original language. But with tag おはようございます.
     • If the recording is too noisy, write in the transcript.

2. Reference Setup

   • Part A: Whisper as reference on the screen.
   • Part B: ATCO2 as reference.
   • Part C: Whisper with a specific prompt. => Question: Shuold we change the Whisper Prompt to Kaldi?

2. Transcription Process

1. Student Transcription

   • Several Japanese students will transcribe the audio.
   • They will receive a random mix of Parts A, B, and C.
   • They are unaware of which reference they are using.
   • Get their TOEIC scores to measure English proficiency.

2. Proofreading

   • An ex-ATC controller proofreads the transcripts. => If possible, get multiple proofreaders to reduce bias.
   • They do not see the reference used, only the student transcriptions.

3. Evaluation & Analysis

1. Accuracy Measurement

   • Compare transcriptions with the proofread versions.
   • Calculate Word Error Rate (WER) for each reference type.

2. Comparison & Insights

   • Identify which reference produces the most accurate transcriptions.
   • Check if certain types of errors are more frequent in specific references.

How to run

get into the container Please write the .env file into the root folder of project DATABASE_URL=mysql+pymysql://user:transcribepass@db/transcribe

cd frontend/
npm install
cd ..
docker compose up --build

Upload the audio file, put audio files into backend\recording folder and frontend\public\recording folder.

# cp the audio files from local to the ssh server
scp -r -i ~/.ssh/atc-transcribe.pem -P 2222 /path/to/local/folder/* ubuntu@54.199.222.244:/path/to/remote/folder

Get into the container of mysql to create the database

docker exec -it atc-transcribe-db-1 bash
mysql -u root -p

Use init.sql to create the database

USE transcribe;
Copy the content of init.sql and paste it into the terminal

docker exec -it atc-transcribe-backend-1 bash
cd backend
python script/import_audio.py
python script/import_transcription.py.py

Sometimes, the .env file is not loaded correctly, so you need to run the following command to load the .env file

docker compose exec frontend env
docker compose exec backend env