asr_meeting_summarization

An end-to-end pipeline for automated transcription and summarization of Vietnamese meeting recordings. The system processes long-duration audio (1–3 hours), identifies individual speakers, generates a time-stamped transcript, and produces structured meeting minutes using a large language model.

---

Overview

This project was built to address a practical problem: long Vietnamese meetings produce hours of audio that are difficult to review manually. The pipeline automates the full workflow — from raw audio file to a polished Markdown meeting report — without requiring manual intervention.

The system is designed to handle real-world conditions such as overlapping silence, multiple speakers, and domain-specific terminology.

---

Pipeline Architecture

The pipeline executes five sequential stages:

Audio File
    |
    v
[Stage 1] Voice Activity Detection + Smart Chunking     (Silero VAD)
    |
    v
[Stage 2] Speaker Diarization                           (pyannote.audio 3.1)
    |
    v
[Stage 3] Speech-to-Text Transcription                  (PhoWhisper-large / Whisper large-v3)
    |
    v
[Stage 4] Transcript Alignment                          (merge diarization + transcription)
    |
    v
[Stage 5] Map-Reduce Summarization                      (Gemini 2.0 Flash / GPT-4o mini)
    |
    v
Output: transcript.json, transcript.txt, meeting_minutes.md

Stage 1 — Voice Activity Detection and Smart Chunking

Uses Silero VAD to detect speech segments and remove silence. Audio is then split into overlapping chunks of approximately 10–15 minutes at natural silence boundaries, ensuring no sentence is cut mid-way. This approach prevents issues with Whisper's fixed 30-second context window on long recordings.

Stage 2 — Speaker Diarization

Uses pyannote/speaker-diarization-3.1 to identify who is speaking at each point in time. The number of speakers can be set manually or detected automatically. Requires a Hugging Face token with access to the pyannote model.

Stage 3 — Transcription

Two backends are supported:

The transcriber processes each audio chunk in 30-second sub-windows aligned to Whisper's input format. Absolute timestamps are preserved across chunks.

Stage 4 — Alignment

Merges diarization segments (who spoke) with transcription segments (what was said) using a time-overlap heuristic. The output is a sequence of AlignedSegment objects, each containing a speaker label, start/end timestamps, and the corresponding text.

Exports:

• transcript.json — machine-readable format
• transcript.txt — human-readable format

Stage 5 — Map-Reduce Summarization

For long meetings, the full transcript exceeds the context limit of most LLMs. The pipeline uses a map-reduce strategy:

1. Map phase — the transcript is split into 15-minute chunks; each chunk is summarized independently by the LLM.
2. Reduce phase — all chunk summaries are combined into a single, structured meeting minutes document in Markdown format.

Two LLM backends are supported: Google Gemini (gemini-2.0-flash) and OpenAI (gpt-4o-mini). The active backend is set in config.py.

---

Project Structure

asr_meeting_summarization/
|
|-- main.py                     # Pipeline entry point
|-- config.py                   # Centralized configuration
|-- requirements.txt
|-- speech-to-text.py           # Standalone transcription script
|
|-- preprocessing/
|   |-- vad_chunker.py          # Stage 1: VAD + smart chunking
|
|-- diarization/
|   |-- speaker_diarizer.py     # Stage 2: Speaker diarization
|
|-- transcription/
|   |-- transcriber.py          # Stage 3: Whisper-based ASR
|
|-- utils/
|   |-- alignment.py            # Stage 4: Transcript + diarization alignment
|
|-- summarization/
|   |-- map_reduce.py           # Stage 5: Map-reduce summarization
|
|-- output/                     # Generated output files (gitignored)
|-- audio_records/              # Input audio files (gitignored)

---

Requirements

• Python 3.10 or higher
• CUDA-capable GPU recommended (CPU inference is supported but slow)
• CUDA 11.8+ (for int8_float16 compute type with faster-whisper)

Python Dependencies

librosa>=0.10.0
soundfile>=0.12.0
faster-whisper>=1.0.0
pyannote.audio>=3.1.0
torch>=2.0.0
torchaudio>=2.0.0
google-generativeai>=0.8.0
python-dotenv>=1.0.0
numpy>=1.24.0

Install all dependencies:

pip install -r requirements.txt

---

Configuration

All settings are managed in config.py. Sensitive credentials are loaded from a .env file and are never committed to the repository.

Create a .env file in the project root:

HF_TOKEN=your_huggingface_token
GEMINI_API_KEY=your_gemini_api_key
OPENAI_API_KEY=your_openai_api_key

Note: A Hugging Face token with accepted terms for pyannote/speaker-diarization-3.1 is required for speaker diarization. Visit the model page on Hugging Face to request access.

Key configuration values in config.py:

---

Usage

Basic Usage

python main.py --input path/to/meeting.mp3

Specify Number of Speakers

python main.py --input path/to/meeting.mp3 --num-speakers 4

Use Fast Multilingual Model

python main.py --input path/to/meeting.mp3 --fast

Skip Diarization (Transcript Only, No Speaker Labels)

python main.py --input path/to/meeting.mp3 --skip-diarization

Skip Summarization (Transcript Only)

python main.py --input path/to/meeting.mp3 --skip-summarization

Custom Output Directory

python main.py --input path/to/meeting.mp3 --output-dir path/to/output/

Full Option Reference

usage: main.py [-h] --input INPUT [--num-speakers NUM_SPEAKERS]
               [--output-dir OUTPUT_DIR] [--skip-diarization]
               [--skip-summarization] [--fast] [--lang LANG]

Options:
  --input, -i          Path to audio file (mp3, wav, flac, etc.)
  --num-speakers, -n   Number of speakers (auto-detect if not set)
  --output-dir, -o     Output directory (default: ./output)
  --skip-diarization   Skip speaker diarization
  --skip-summarization Skip LLM summarization
  --fast               Use openai/whisper-large-v3 instead of PhoWhisper
  --lang               Language code override (e.g. vi, en)

---

Output Files

After a successful run, the following files are written to the output directory:

Example Transcript Format (transcript.txt)

[00:00:12] SPEAKER_00: Xin chào tất cả mọi người, hôm nay chúng ta sẽ thảo luận về kế hoạch quý 2.
[00:00:28] SPEAKER_01: Vâng, tôi đã chuẩn bị báo cáo tình hình dự án.

Example Meeting Minutes Format (meeting_minutes.md)

# BIEN BAN CUOC HOP

## Thong tin chung
| Thong tin    | Chi tiet         |
|---|---|
| Thoi luong   | ~1 gio 45 phut   |
| So nguoi     | 4                |

## Tom tat tong quan
...

## Action Items
| STT | Nhiem vu | Nguoi phu trach | Ghi chu |
|---|---|---|---|
...

---

Standalone Transcription Script

For cases where only transcription is needed (without diarization or summarization), a lightweight standalone script is available:

python speech-to-text.py

This script loads an audio file and outputs raw transcriptions using PhoWhisper-large, without the full pipeline overhead.

---

Performance Notes

• On a GPU (e.g., RTX 3080), a 1-hour meeting typically processes in 15–25 minutes end-to-end.
• On CPU only, expect 2–4x realtime for transcription. Diarization is the most GPU-dependent stage.
• The --fast flag uses whisper-large-v3 which is faster on CPU but slightly less accurate for Vietnamese.
• VAD significantly reduces transcription time by removing silence before processing.

---

License

This project is intended for academic and research purposes. Third-party models used in this pipeline (PhoWhisper, pyannote.audio) are subject to their respective licenses. Refer to their respective Hugging Face model pages for terms of use.

---

Acknowledgements

• VinAI Research — PhoWhisper Vietnamese ASR model
• pyannote.audio — Speaker diarization framework
• Silero VAD — Lightweight voice activity detection
• OpenAI Whisper — Multilingual speech recognition
• Google Gemini — LLM summarization backend