PycergAI Benchmark

Open benchmarking toolkit for evaluating NLP models on low-resource Burkinabe languages. Developed as part of a UNICEF consultancy on NLP for linguistic inclusion in West and Central Africa. Currently focuses on Moore and Dioula across three evaluation tasks: MT, ASR, and TTS.

---

Supported Languages

Supported Tasks

---

Architecture

PycergBench uses a provider-based architecture inspired by the companion ai-service project. Every model or API is wrapped in a provider that implements a standard interface (MTProvider, ASRProvider, TTSProvider). The evaluators are completely model-agnostic.

providers/
  base.py          # Abstract interfaces: MTProvider, ASRProvider, TTSProvider
  factory.py       # Resolves a model_id to the right provider
  api.py           # HTTP API providers (for Docker microservices)
  mt/hf_seq2seq.py # HuggingFace seq2seq models (NLLB, mBART, ...)
  asr/hf_pipeline.py # HuggingFace ASR pipeline (Whisper, MMS, ...)
  tts/hf_tts.py    # HuggingFace TTS models (MMS-TTS, VITS, ...)

To add a new model backend, implement the relevant abstract class and register it in the factory.

---

Setup

git clone https://github.com/pycerg-ai/pycergai-benchmark.git
cd pycergai-benchmark

python -m venv .venv
.venv\Scripts\activate       # Windows
pip install -e .

# Optional: TTS naturalness scoring
pip install git+https://github.com/sarulab-speech/UTMOSv2.git

HuggingFace Token (required for gated datasets)

Create a .env file at the repo root:

HF_TOKEN=hf_your_token_here

Or set the environment variable directly:

export HF_TOKEN=hf_your_token_here

---

Dataset Configuration

Datasets are configured in configs/datasets/<language>.yaml. Each source specifies the HF repo, column mappings, and benchmark groups -- named subsets of splits to evaluate together.

language: mos_Latn

tasks:
  mt:
    sources:
      - id: goai_flores_fra_mos
        type: huggingface
        hf_dataset: goaicorp/flores-plus-fra-mos
        source_column: sentence_fra_Latn
        target_column: sentence_mos_Latn
        source_lang: fra_Latn
        target_lang: mos_Latn
        benchmark_groups:
          general:
            splits: [test, devtest]
            description: "FLORES+ general knowledge (test + devtest)"
          health:
            splits: [health]
          education:
            splits: [education]
          child_protection:
            splits: [child_protection]
          environment:
            splits: [environment]

Default Benchmark Datasets

Moore (mos_Latn)

Dioula (dyu_Latn)

---

Running Benchmarks

By default (no --split or --group), the script evaluates all benchmark groups defined in the YAML config. The model is loaded once and reused across all groups.

MT: French -> Moore (all groups)

Evaluates general knowledge and all 4 domains in a single command:

python scripts/run_benchmark.py \
  --task mt \
  --language mos_Latn \
  --model facebook/nllb-200-distilled-600M \
  --no-comet \
  --verbose

MT: Single group only (e.g. health)

python scripts/run_benchmark.py \
  --task mt \
  --language mos_Latn \
  --model facebook/nllb-200-distilled-600M \
  --group health \
  --no-comet \
  --verbose

MT: Moore -> French (reverse direction, all groups)

python scripts/run_benchmark.py \
  --task mt \
  --language mos_Latn \
  --model facebook/nllb-200-distilled-600M \
  --source-lang mos_Latn \
  --target-lang fra_Latn \
  --source-column sentence_mos_Latn \
  --target-column sentence_fra_Latn \
  --no-comet \
  --verbose

MT: Reverse direction, single group

python scripts/run_benchmark.py \
  --task mt \
  --language mos_Latn \
  --model facebook/nllb-200-distilled-600M \
  --source-lang mos_Latn \
  --target-lang fra_Latn \
  --source-column sentence_mos_Latn \
  --target-column sentence_fra_Latn \
  --group general \
  --no-comet \
  --verbose

ASR: Moore (all groups)

python scripts/run_benchmark.py \
  --task asr \
  --language mos_Latn \
  --model openai/whisper-large-v3 \
  --verbose

TTS: Moore (all groups)

python scripts/run_benchmark.py \
  --task tts \
  --language mos_Latn \
  --model facebook/mms-tts-mos \
  --verbose

MT: French -> Dioula (all groups)

python scripts/run_benchmark.py \
  --task mt \
  --language dyu_Latn \
  --model facebook/nllb-200-distilled-600M \
  --no-comet \
  --verbose

MT: Dioula -> French (reverse direction, all groups)

python scripts/run_benchmark.py \
  --task mt \
  --language dyu_Latn \
  --model facebook/nllb-200-distilled-600M \
  --source-lang dyu_Latn \
  --target-lang fra_Latn \
  --source-column sentence_dyu_Latn \
  --target-column sentence_fra_Latn \
  --no-comet \
  --verbose

ASR: Dioula (all groups)

python scripts/run_benchmark.py \
  --task asr \
  --language dyu_Latn \
  --model openai/whisper-large-v3 \
  --verbose

TTS: Dioula (all groups)

python scripts/run_benchmark.py \
  --task tts \
  --language dyu_Latn \
  --model facebook/mms-tts-dyu \
  --verbose

Using an API backend

python scripts/run_benchmark.py \
  --task mt \
  --language mos_Latn \
  --model api:http://localhost:5003 \
  --source-lang fra_Latn \
  --target-lang mos_Latn

Using an explicit split (bypasses benchmark groups)

python scripts/run_benchmark.py \
  --task mt \
  --language mos_Latn \
  --model facebook/nllb-200-distilled-600M \
  --split test \
  --no-comet \
  --verbose

---

Output Structure

Results are saved in a clean hierarchy:

results/
  mos_Latn/
    mt/
      fra_Latn_mos_Latn/          # MT: source_target (avoids clash with reverse or other langs)
        nllb-200-distilled-600M/
          general.json
          health.json
          ...
          summary.json
        comparison.json
        comparison.md
      mos_Latn_fra_Latn/          # reverse direction
        ...
    asr/
      whisper-large-v3/
        general.json
        summary.json
      comparison.json
      comparison.md
    tts/
      mms-tts-mos/
        general.json
        summary.json
  dyu_Latn/
    mt/
      fra_Latn_dyu_Latn/
        ...

Paths use CLI --language first, then task. MT adds <source_lang>_<target_lang>. Each JSON includes benchmark_language and (for MT) mt_pair.

• Per-group files: metrics per benchmark group; summary.json: all groups + groups_evaluated.
• comparison.*: model ranking; headers list task, benchmark language, MT pair (if MT), and group columns.

Each JSON contains metrics, model info, and a UTC timestamp. When a group is re-evaluated, the file is overwritten (latest run = reference).

Generating Model Comparisons

After evaluating multiple models, generate (or refresh) comparison tables:

# Compare all MT models for Moore
python scripts/compare_results.py --language mos_Latn --task mt

# Compare all ASR models for Dioula
python scripts/compare_results.py --language dyu_Latn --task asr

# Compare all tasks for a language
python scripts/compare_results.py --language mos_Latn --task all

This reads existing result files and produces comparison.json + comparison.md in results/<language>/<task>/.

---

Full CLI Reference

python scripts/run_benchmark.py [OPTIONS]

Options:
  --task            [mt|asr|tts]          Evaluation task (required)
  --language        TEXT                  HF language code (required)
  --model           TEXT                  Model ID, path, or api:<url> (required)
  --source-lang     TEXT                  Source language (MT only)
  --target-lang     TEXT                  Target language (MT only)
  --source-column   TEXT                  Override source column name (MT)
  --target-column   TEXT                  Override target column name (MT)
  --dataset         TEXT                  Explicit source: hf:<id> or local:<path>
  --split           TEXT                  HF dataset split (bypasses groups)
  --group           TEXT                  Single benchmark group to evaluate
  --output          TEXT                  Root output directory [default: results]
  --batch-size      INTEGER              [default: 8]
  --max-samples     INTEGER              Limit samples per group
  --device          [cpu|cuda|mps|auto]  [default: auto]
  --submit-to-leaderboard                Append to leaderboard
  --loopback-asr-model TEXT             TTS loopback model
  --no-comet                             Skip COMET (MT)
  --verbose                              Debug logging
  --version                              Show version

---

Adding a Custom Provider

1. Implement the abstract class for your task (e.g. MTProvider).
2. Register your provider in providers/factory.py or pass it directly via the Python API.

from pycergai_benchmark.providers.base import MTProvider, ProviderInfo
from pycergai_benchmark.tasks.mt import MTEvaluator

class MyCustomMT(MTProvider):
    def translate(self, text, source_lang, target_lang):
        return my_custom_translate(text, source_lang, target_lang)

    def info(self):
        return ProviderInfo(name="my_custom", model_id="custom/v1")

provider = MyCustomMT()
evaluator = MTEvaluator(provider, "fra_Latn", "mos_Latn")
result = evaluator.evaluate(data)

---

Python API

from pycergai_benchmark.core.config_loader import ConfigLoader
from pycergai_benchmark.core.data_loader import DataLoader
from pycergai_benchmark.providers.factory import create_mt_provider
from pycergai_benchmark.tasks.mt import MTEvaluator

cfg = ConfigLoader()
loader = DataLoader()

source_cfg = cfg.get_dataset_source("mos_Latn", "mt")
data = loader.load_from_config(source_cfg, "mt", split="test")

provider = create_mt_provider("facebook/nllb-200-distilled-600M")
evaluator = MTEvaluator(provider, "fra_Latn", "mos_Latn")
result = evaluator.evaluate(data, compute_comet=False)
print(f"chrF++: {result.overall_chrf:.1f}")
print(f"BLEU:   {result.overall_bleu:.1f}")

---

Citation

@misc{pycergai_benchmark_2026,
  title={PycergAI Benchmark: Open Evaluation Platform for
         West and Central African Language Models},
  author={PycergAI Benchmark Contributors},
  year={2026},
  howpublished={\url{https://github.com/pycerg-ai/pycergai-benchmark}},
  note={Developed under UNICEF WCARO consultancy on NLP
        for linguistic inclusion}
}

Acknowledgments

• UNICEF West and Central Africa Regional Office (WCARO)
• Masakhane -- NLP research community for Africa
• FLORES-200 / FLORES+ (Meta AI) -- multilingual evaluation datasets
• GO AI Corp -- Moore and Dioula benchmark datasets
• AfricaNLP community and workshop contributors

License

Apache 2.0 -- see LICENSE.