OpenWHO: A Document-Level Parallel Corpus for Health Translation in Low-Resource Languages

📝 Paper | 📊 Data

This repository contains data extraction and evaluation code for the OpenWHO health MT evaluation corpus.

Getting started

1. Install requirements with uv sync
2. Prepare a .env file with secrets:

OPENROUTER_API_KEY=...
HF_TOKEN=...
GOOGLE_API_KEY=...

Data preparation

Code for scraping, filtering, and publishing data to the HuggingFace repo is in the dataprep directory.

Evaluation

Code for evaluating model perf on OpenWHO and WMT24++ is in the eval directory:

• metric_openwho.py: Script for computing eval metrics (ChrF / MetricX).
• eval_openwho.py: Script for running inference on NMT models (NLLB / MADLAD)
• eval_gemini_sentlevel.py: Script for running LLM inference at the sent level (on OpenWHO / WMT24++, with Gemini / Deepseek / Gemma3).
• eval_gemini_sent_window.py: LLM inference with a sliding context window around the sentence to translate.
• eval_gemini_sent_context.py: LLM inference with the whole document context, but translating one sentence at a time.
• eval_gemini_doclevel.py: Script for running LLM inference at the doc level. Argument --translate-again is used to translate again.