💊 DocTalkBN: A Novel Dataset of Expert Telemedicine Conversations in Bengali

DocTalkBN is an expert-grounded, multimodal dataset of real-world doctor-patient interactions in Bengali, sourced from health programs on Bengali television and YouTube channels. It captures authentic clinical conversations across 25+ medical specialties and provides curated benchmarks for four downstream medical NLP tasks.

Language: Bengali (Bangla)  |  Domain: Healthcare / Clinical NLP  |  Platform: Linux

---

Example Dataset

The example_dataset/ directory contains a small, ready-to-use sample of the full corpus — one folder per video, each named by its YouTube video ID.

example_dataset/
└── <video_id>/                                          # one entry per video (YouTube ID)
    ├── <video_id>_yt-dlp-metadata.json                  # raw YouTube metadata (title, channel, duration, …)
    ├── <video_id>_derived-metadata.json                 # derived fields: medical tags, specialty, etc.
    ├── audio/
    │   └── <video_id>_audio.mp3                         # downloaded audio
    └── transcribed/
        └── yt-auto/                                     # source: YouTube auto-captions
            ├── <video_id>_transcription.srt             # raw SRT captions
            ├── <video_id>_transcription-processed-with-timestamp.txt
            ├── <video_id>_transcription-processed-without-timestamp.txt
            └── parsed/
                └── gemini-3-flash-preview/
                    └── <video_id>_conversation.json     # structured conversation (exchanges + turns)

The conversation.json at the leaf is the core artifact — a list of exchanges (type host_doctor_qa or patient_call), each containing timestamped, speaker-labelled turns. See the Dataset Format section for the schema.

---

Overview

Access to reliable medical information remains a challenge for Bengali-speaking populations. Existing medical NLP resources are predominantly English-centric, text-only, or lack grounding by domain experts. DocTalkBN addresses this gap by:

• Collecting real doctor-patient conversations from Bengali TV channels and health programs
• Processing audio through automatic speech recognition, speaker diarization, and LLM-based transcription parsing
• Curating three downstream task datasets from the resulting conversations
• Benchmarking LLMs, BERT-based models, and embedding-based classifiers on each task

The dataset captures code-mixed Bengali, regional medical nuances, and multi-turn conversational dynamics that are absent from synthetic or translated corpora.

---

Downstream Tasks

---

Repository Structure

doctalk/
├── src/                            # Main source code
│   ├── constants.py                # Medical specialties, program names
│   ├── search_yt_videos_*.py       # YouTube video search & scraping
│   ├── filter_healthcare_data.py   # LLM-based healthcare video filtering
│   ├── fetch_metadata_and_process_transcriptions.py
│   ├── extract_tag_and_derived_metadata.py
│   ├── parse_transcriptions.py     # SRT/VTT → structured conversations
│   ├── generate_medical_ner_dataset.py
│   ├── generate_triage_dataset.py
│   ├── generate_advice_safety_dataset.py
│   ├── organize_downstream_datasets.py
│   ├── infer_downstream.py         # Unified inference script (all tasks)
│   ├── run_bulk_inference.py       # Bulk multi-model inference
│   ├── prompts/                    # Prompt templates (28 task-specific dirs)
│   ├── downstream-artifacts/       # Fine-tuning & inference code
│   │   ├── medical-ner/            # BanglaBERT / mmBERT NER fine-tuning
│   │   ├── triage/                 # Triage classification fine-tuning
│   │   └── advice-safety/          # Advice safety fine-tuning
│   ├── evaluation-pipeline/        # Evaluation metrics & analysis
│   ├── dataset-statistics-generator/
│   └── saved-data/                 # Generated datasets & checkpoints
├── audio-analysis/
│   ├── bangla-asr/                 # ASR experiments (Whisper, Wav2Vec2, etc.)
│   └── diarization/                # Speaker diarization (NeMo, PyAnnote, 3D-Speaker)
├── llm-inference/
│   ├── api/                        # Gemini API audio inference
│   └── local/                      # Local model inference
├── examples/                       # Sample conversations & dataset entries
├── workflow/                       # Step-by-step pipeline documentation
└── data/                           # Downloaded video metadata & IDs

---

Prerequisites

pip install -U yt-dlp
pip install -U torch transformers datasets evaluate seqeval accelerate
pip install -U sentence-transformers
pip install -U scikit-learn
pip install -U git+https://github.com/csebuetnlp/normalizer.git  # BanglaBERT only

---

Pipeline

Step 1 — Video Search & Metadata Collection

cd src/

# Search a single program
python search_yt_videos_single_query.py \
  --query "স্বাস্থ্য জিজ্ঞাসা - Bangladesh Television" \
  --channel UClzwimpLoZu9us9MwalLbtA \
  --data-dir saved-data \
  --file results-btv.json

# Bulk search across all configured programs
python search_yt_videos_bulk.py \
  --queries-json search_queries.json \
  --data-dir saved-data

Step 2 — Healthcare Video Filtering

python filter_healthcare_data.py \
  --folder saved-data \
  --file results.json \
  --model qwen3:30b-instruct

Step 3 — Metadata Extraction & Transcription Fetching

python fetch_metadata_and_process_transcriptions.py \
  --folder saved-data \
  --file filtered-results.json \
  --lang bn \
  --force-rewrite

python extract_tag_and_derived_metadata.py \
  --folder saved-data \
  --file filtered-results.json \
  --model qwen3:30b-instruct

Step 4 — Transcription Parsing (SRT → Structured Conversations)

python parse_transcriptions.py \
  --folder saved-data \
  --file filtered-results.json \
  --model gemini-3-flash-preview

# Test on a single video
python parse_transcriptions.py \
  --folder saved-data \
  --file filtered-results.json \
  --model gemini-3-flash-preview \
  --test-video-id RkWh5fOOx9s

Step 5 — Downstream Dataset Generation

# Medical NER
python generate_medical_ner_dataset.py --model gemini-3-flash-preview --batch-size 5

# Triage Classification
python generate_triage_dataset.py

# Advice Safety
python generate_advice_safety_dataset.py

Step 6 — Organize into Train / Val / Test Splits

# All tasks with default 80/10/10 split
python organize_downstream_datasets.py

# Custom split
python organize_downstream_datasets.py --train-pct 80 --val-pct 10

# Single task
python organize_downstream_datasets.py --tasks medical-ner

# Multiple tasks (semicolon-separated)
python organize_downstream_datasets.py --tasks "medical-ner;advice-safety;triage"

---

Inference

LLM Inference (Zero-Shot / Few-Shot / Chain-of-Thought)

cd src/

# Zero-shot inference
python infer_downstream.py \
  --tasks triage \
  --split test \
  --model gemini-2.5-flash \
  --setting zero-shot \
  --batch-size 3

# Chain-of-Thought (CoT) inference
python infer_downstream.py \
  --tasks triage \
  --split test \
  --model gemini-2.5-flash \
  --setting cot \
  --batch-size 3

# Multiple tasks in one run (semicolon-separated)
python infer_downstream.py \
  --tasks "medical-ner;advice-safety;triage" \
  --split test \
  --model gemini-2.5-flash \
  --setting zero-shot \
  --batch-size 3

# OpenAI GPT-4O
python infer_downstream.py \
  --tasks advice-safety \
  --split test \
  --model gpt-4o \
  --provider openai \
  --setting zero-shot \
  --batch-size 3

# OpenRouter (Qwen, LLaMA, etc.)
python infer_downstream.py \
  --tasks medical-ner \
  --split test \
  --model qwen/qwen3-32b \
  --provider openrouter \
  --setting zero-shot \
  --batch-size 3

Bulk Inference (Multiple Models)

python run_bulk_inference.py --tasks medical-ner --split test --batch-size 3
python run_bulk_inference.py --tasks "medical-ner;advice-safety;triage" --split test --batch-size 3

infer_downstream.py — Key Parameters

---

Fine-Tuning

Medical NER — BanglaBERT / mmBERT

cd src/downstream-artifacts/medical-ner/ner-finetuning

# BanglaBERT (~110M parameters)
python train_ner.py --model banglabert

# BanglaBERT — small dataset config (~38 samples)
python train_ner.py --model banglabert \
  --epochs 30 --train-batch 4 --grad-accum 2 \
  --warmup-steps 20 --early-stopping 7 --logging-steps 10

# mmBERT (~307M parameters)
python train_ner.py --model mmbert

# mmBERT — small dataset config with FP16
python train_ner.py --model mmbert \
  --epochs 30 --train-batch 2 --grad-accum 4 \
  --warmup-steps 20 --early-stopping 7 --fp16

# Standalone NER inference
python infer_ner.py --model banglabert --text "রোগীর হিট স্ট্রোক, জ্বর এবং সর্দি আছে।"
python infer_ner.py --model mmbert    --text "রোগীর হিট স্ট্রোক, জ্বর এবং সর্দি আছে।"

Triage Classification — BERT-based & Embedding-based

cd src/downstream-artifacts/triage/triage-finetuning

# BERT sequence classifiers
python train_triage.py --model banglabert
python train_triage.py --model mmbert --fp16

# Embedding + MLP classifiers (frozen encoder)
python train_triage_embed.py --model multilingual-minilm
python train_triage_embed.py --model multilingual-e5-small

# Standalone inference
python infer_triage.py --model banglabert --text "রোগীর হিট স্ট্রোক, জ্বর এবং সর্দি আছে।"
python infer_triage.py --model multilingual-minilm --json-input input.json --json-output preds.json

Advice Safety — Embedding-based Pair Classifier

cd src/downstream-artifacts/advice-safety/advice-safety-finetuning

python train_advice_safety_embed.py --model multilingual-minilm
python train_advice_safety_embed.py --model multilingual-e5-small

# Standalone inference
python infer_advice_safety.py --model multilingual-minilm \
  --json-input input.json --json-output preds.json

Fine-Tuned Model Evaluation via Unified Pipeline

cd src/

# Medical NER
python infer_downstream.py --tasks medical-ner --split test --model banglabert --setting finetuned
python infer_downstream.py --tasks medical-ner --split test --model mmbert      --setting finetuned

# Triage
python infer_downstream.py --tasks triage --split test --model banglabert            --setting finetuned
python infer_downstream.py --tasks triage --split test --model multilingual-minilm   --setting finetuned
python infer_downstream.py --tasks triage --split test --model multilingual-e5-small --setting finetuned

# Advice Safety
python infer_downstream.py --tasks advice-safety --split test --model multilingual-minilm   --setting finetuned
python infer_downstream.py --tasks advice-safety --split test --model multilingual-e5-small --setting finetuned

---

Models & Baselines

Closed-Source LLMs

Open-Source LLMs (via OpenRouter / Local)

BERT-type & Embedding Models

---

Data Sources

DocTalkBN aggregates health program content from 14 major Bengali TV channels:

Medical specialties covered include cardiology, neurology, gastroenterology, pediatrics, psychiatry, orthopedics, dermatology, endocrinology, and 17 others.

---

Dataset Format

Each processed video is stored as a JSON array of conversation exchanges. Each exchange has a type, a timestamp marking its position in the video, and a turns list of speaker utterances.

Two exchange types are distinguished:

• host_doctor_qa — a discussion between the show host and the doctor (general medical Q&A)
• patient_call — a live phone-in patient consultation with the doctor

[
  {
    "type": "host_doctor_qa",
    "timestamp": "00:01:15.910",
    "turns": [
      { "speaker": "host",   "text": "মাথাব্যথা কখন সতর্কবার্তা?" },
      { "speaker": "doctor", "text": "৯০–৯৫% ক্ষেত্রে মাথাব্যথা সাধারণ টেনশন টাইপ..." }
    ]
  },
  {
    "type": "patient_call",
    "timestamp": "00:12:12.389",
    "turns": [
      { "speaker": "patient", "text": "আমার হঠাৎ হঠাৎ মাথা ব্যথা হয়, বয়স ৪৮।" },
      { "speaker": "doctor",  "text": "আপনার অন্য কোনো সমস্যা আছে?" },
      { "speaker": "patient", "text": "না, অন্য কোনো সমস্যা নাই।" },
      { "speaker": "doctor",  "text": "স্থানীয় একজন ফিজিশিয়ানকে দেখান..." }
    ]
  }
]

---

Evaluation

Evaluation scripts are in src/evaluation-pipeline/.

cd src/evaluation-pipeline/

# Triage classification metrics (per-class + macro/micro F1)
python triage-results.py

# NER evaluation (BIO format, seqeval-compatible)
python ner-eval.py

# Advice safety metrics
python main.py

See src/evaluation-pipeline/EVALUATION_METRICS.md for detailed metric definitions.

---

Future

We are actively working on the Advice Generation task, where a model receives only the patient profile and generates a free-text medical recommendation. This will include LLM-as-a-judge evaluation alongside automatic metrics (BLEU, token overlap). Stay tuned for updates.

---

---

License

Please refer to the repository for license information.