Proverbs Run in Pairs: Evaluating Proverb Translation Capability of Large Language Models

Paper: https://arxiv.org/abs/2501.11953

---

📌 Overview

Proverbs are culturally rich, figurative expressions whose meanings often diverge from their literal wording. While modern machine translation (MT) systems excel at literal and contextual translation, their ability to translate culturally grounded proverbs remains largely unexplored.

This repository provides:

• Two new datasets for evaluating proverb translation:

  • Standalone Proverb Translation across four language pairs.
  • Proverbs-in-Conversation (PiC) mined from movie subtitles.

• Evaluation code for:

  • Open-source LLMs,
  • NLLB/Seq2Seq MT systems,
  • GPT-4o-mini API,
  • GPT-4o-mini based win-rate evaluation.

---

📂 Repository Structure

curated_data/
   proverb_translation/
       en_de.csv, de_en.csv, en_zh.csv, ...
   subtitle_translation/
       en_de.en.json, en_de.de.json, en_xx.json, ...

raw_data/
   MAPS_Final/     # Original MAPS dataset used as seed

scripts/
   evaluate_proverb.sh
   evaluate_subtitle.sh
   eval_winrate.sh

src/
   evaluate_proverb_openai.py
   evaluate_proverb_openllm.py
   evaluate_proverb_seq2seq.py
   evaluate_subtitle_openai.py
   evaluate_subtitle_openllm.py
   evaluate_subtitle_seq2seq.py
   eval_winrate.py

---

1. Dataset Descriptions

1.1 Standalone Proverb Translation Dataset

We extend the MAPS dataset (Liu et al., 2024) by providing high-quality human-verified translation pairs for:

• English (EN)
• German (DE)
• Bengali (BN)
• Indonesian (ID)
• Mandarin Chinese (ZH)

Data format (*.csv)

Each file contains:

Example:

,proverb,translation
0,水能载舟，亦能覆舟,"Water can carry a boat, but it can also overturn it"
1,失败乃成功之母,Failure is the mother of success
2,擒贼先擒王,"You need to catch the king first if you want to catch the thief"

There are 8 translation directions, covering 4 languages ↔ English.

---

1.2 Proverb-in-Conversation Dataset (PiC)

This dataset contains naturally occurring proverb usage mined from OpenSubtitles. For each proverb, we include:

• Source sentence containing the proverb
• Extracted translation from subtitles
• Movie metadata
• Left/right contextual sentences
• Quality estimation (LLM-QE & DA-QE)
• Human verification annotations

Data format (*.json)

Each file contains a list of mined proverb usage records:

{
  "proverb": "水能载舟，亦能覆舟",
  "pairs": [
    {
      "id": "1804934",
      "meta": {
        "year": 1994,
        "subtitleId": {"en": 6678343, "zh": 5958744},
        "detail": {"title": "Drunken Master II", "languages": ["Cantonese","Japanese"]}
      },
      "translation": {
        "en": "\"Boats can float on water...\"",
        "zh": "要知道水能载舟 亦能覆舟"
      },
      "left_adjacent": {...},
      "right_adjacent": {...},
      "similarity": 1.0,
      "llm_verify_prvb_in_src": true,
      "overall_score": 25.8,
      "src_lang": "zh",
      "tgt_lang": "en"
    }
  ]
}

Some proverbs have multiple mined pairs.

---

2. Research Contribution Summary

1️⃣ We build a cross-lingual proverb translation benchmark

• Covers 4 languages
• Includes both isolated proverb translation and proverb-in-conversation

2️⃣ We show LLMs outperform NMT models on proverb translation

• Especially for figurative proverbs
• Especially when languages share similar cultural regions
• GPT-4o-mini and Llama-3.1-70B show the strongest performance**

3️⃣ We show standard MT metrics fail on figurative language

• BLEU, CHRF++, COMET are over-sensitive to lexical choices
• They fail even when meaning is semantically preserved
• LLM-as-Judge helps, but still imperfect

4️⃣ We analyze prompt & context effects

• Adding proverb explanations has no benefit
• Dialogue-style context greatly improves accuracy
• LLMs already encode most proverb semantics in pretraining

---

3. Using the Code

3.1 Installation

A requirements.txt file will be added. For now:

pip install -r requirements.txt

---

3.2 Running Experiments

▶️ 1. Proverb Translation (Standalone)

bash scripts/evaluate_proverb.sh

This runs:

• Seq2Seq baselines (NLLB family)
• Open-source LLMs
• GPT-4o-mini API (if key is provided)

---

▶️ 2. Proverbs in Conversation Translation

bash scripts/evaluate_subtitle.sh

Supports:

• Zero-shot / one-shot prompting
• Proverb explanation prompts
• Dialogue-context prompts
• Concatenation-context prompts

---

▶️ 3. Win-Rate Evaluation (GPT-4o-mini)

bash scripts/eval_winrate.sh

This compares two system outputs based on:

• Translation accuracy
• Fluency
• Cultural appropriateness

---

4. Citation

If you use this dataset or code, please cite:

@misc{wang2025proverbsrunpairsevaluating,
      title={Proverbs Run in Pairs: Evaluating Proverb Translation Capability of Large Language Model},
      author={Minghan Wang and Viet-Thanh Pham and Farhad Moghimifar and Thuy-Trang Vu},
      year={2025},
      eprint={2501.11953},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

And the MAPS dataset (source of seed proverbs):

@misc{liu2024multilingualllmsculturallydiversereasoners,
      title={Are Multilingual LLMs Culturally-Diverse Reasoners?},
      author={Chen Cecilia Liu and Fajri Koto and Timothy Baldwin and Iryna Gurevych},
      year={2024},
      eprint={2309.08591},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}