s2st — Speech-to-Speech Translation

P13 of the NLP-in-Industry final assignment · Le Dinh Minh Quan (student 23127460)

Translate spoken input in one language into spoken output in another, delivered as a controllable cascade: pretrained Whisper ASR → a trainable machine-translation core (the NLP heart) → pretrained SpeechT5 TTS. A deterministic agent (FSM, decisions D1–D5) gates quality at every hop — ASR-quality, round-trip verification, audio-QA — and flags low-confidence output for human review rather than shipping it silently. Default direction French → English (configurable). The cascade is chosen over an end-to-end model because it is debuggable, error-attributable, and license-clean; facebook/seamless-m4t-v2-large is supported as an optional end-to-end engine (CC-BY-NC).

Runs fully offline (text passthrough ASR + dictionary MT + placeholder TTS + pure-python metrics) and upgrades to the real models when their libraries + weights are present.

Assignment requirements → how delivered

The cascade (and what's trainable)

audio in -> [D1 route] -> ASR (Whisper, frozen) -> [D2 quality gate]
         -> MT (TRAINABLE core, fine-tuned) -> [D3 round-trip verify]
         -> TTS (SpeechT5, frozen) -> [D4 audio-QA gate] -> [D5 confidence] -> audio out

The only fine-tuned stage is the MT model — the NLP heart. ASR + TTS are strong pretrained models.

Repo layout

13_Speech_to_Speech_Translation/
├── src/s2st/
│   ├── config.py  cli.py  logging_utils.py
│   ├── data/        # dataset (covost2/opus) + samples (fr->en seed + dictionary) + download
│   ├── models/      # model registry
│   ├── asr/         # Whisper transcriber + text passthrough
│   ├── mt/          # THE CORE: transformer translator (m2m100/NLLB) + dictionary baseline
│   ├── tts/         # SpeechT5 synthesizer + placeholder fallback + WAV writer
│   ├── training/    # train_mt + train_baseline + evaluate + tune + metrics (chrF/BLEU/WER)
│   ├── agent/       # state + policy (D1-D5) + tools + llm_orchestrator + s2st_agent (FSM)
│   ├── api/         # schemas + dependencies + main (FastAPI) + ui (Gradio) + app_combined
│   ├── analysis/    # error_analysis + latency
│   ├── autoreport/  # artifact_loader + charts + report_pdf + slides_pptx
│   ├── monitoring/  # drift_report (operational monitor-log)
│   ├── automation/  # autopilot (one button)
│   └── grading/     # checklist (rubric self-check)
├── configs/  data/  models/  tests/  docs/ (15 md)  notebooks/  app/  deploy/  scripts/  sample_data/
├── Dockerfile  docker-compose.yml  Makefile  pyproject.toml  requirements*.txt  .github/workflows/ci.yml

Data & models

Commercial-clean default stack = m2m100 (MIT) + SpeechT5 (MIT) + Whisper (Apache). See docs/data_card.md and docs/model_card.md.

Quickstart (offline, no GPU, no downloads)

pip install -e .                 # core only (numpy/sklearn/pyyaml/pydantic) — no torch
export PYTHONPATH=src

s2st train-baseline              # dictionary MT baseline (instant, CPU)
s2st evaluate --fast             # chrF/BLEU: model vs dictionary vs identity floor
s2st demo-agent --fast           # run the cascade agent on the seed pairs (D1-D5)
s2st translate --text "Le president a signe un nouvel accord commercial."

Add the full stack: pip install -e ".[all]".

The agent (decisions D1–D5)

route (D1) → transcribe (ASR, D2 quality gate) → translate (MT) → verify (D3 round-trip) → synthesize (TTS, D4 audio-QA) → finalize (D5 confidence) — uniform tool contract, ToolTrace audit, optional LLM QA-note brain (OFF by default). Fail-soft: low-confidence output is flagged for human review, never shipped as final. See docs/agent_architecture.md.

Train on Colab (H100, auto-adapts A100/L4/T4)

Open notebooks/S2ST_Colab_Training_H100_AUTOPILOT.ipynb, set the controls, Run all. Resume-safe, GPU-auto-profiled, Colab-safe install. Walkthrough: notebooks/COLAB_GUIDE.md.

One-button autopilot

s2st autopilot     # data → baseline → train-mt → evaluate → tune → analysis → report.pdf + slides.pptx + grade + bundle

Serve

s2st serve --ui    # FastAPI at :8000 (+ Gradio demo at /ui)
# POST /translate-text  {text, synth?}        -> translation (+ audio)
# POST /translate-speech (audio upload)        -> transcript + translation + target audio
# POST /transcribe (audio)                     -> transcript

Tests

pip install -e ".[dev]" && pytest -q     # CPU-only, offline, no downloads

Evaluation

chrF (headline) + BLEU on the MT output; ASR-BLEU for the full speech→speech pipeline (synthesize → re-ASR → BLEU); WER for the ASR stage. The fine-tuned MT core must beat the zero-shot / dictionary / identity baselines on chrF. See docs/evaluation.md.

License

MIT (code). Datasets/models keep their own licenses — see the flags above. The default stack is commercial-clean; CoVoST 2 / NLLB / MMS-TTS / SeamlessM4T are non-commercial.