Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment

🔖 Overview

This work explores a novel perspective: leveraging the preference knowledge embedded within existing English-aligned models to facilitate multilingual preference alignment. Prior work has demonstrated that the implicit reward model, derived from the logits of a well-aligned English DPO model and its reference model, effectively captures preferences over English instructions.
Building on this, we apply this implicit reward model to the multilingual setting, using it to label preference relations in cross-lingual instruction-response pairs. This ensures that multilingual responses are evaluated based on their alignment with English instructions. We term Implicit Cross-Lingual Rewarding, which preserves reward signal fidelity by directly evaluating multilingual responses under English instructions, avoiding translation-induced distortions.

Our approach involves three key steps:

• Multilingual response generation: Starting from any multilingual model that is DPO-tuned on English preference data from an initial reference model. Responses are sampled by the model from multilingual prompts.

• Implicit cross-lingual rewarding: Constructing cross-lingual instruction-response pairs by pairing English instructions with sampled multilingual responses. The implicit reward model then assigns preference scores to these responses, capturing the model's learned preference knowledge.

• Preference Transfer Training: Our approach adopts iterative DPO similar to previous works, incorporating a negative log-likelihood (NLL) loss term to train on the multilingual preference data, thereby transferring preferences across languages.

🔥 Update

• [25/08/2025] We have released all checkpoints used in our work on 🤗HuggingFace. Apologies for the delay!
• [23/04/2025]🍉Implicit Cross-lingual Rewarding has been accepted by ACL 2025!
• [06/03/2025]🔥We release the code for Implicit Cross-Lingual Rewarding!
• [05/03/2025]🔥Implicit Cross-Lingual Rewarding is coming! We release the paper!

📃 Contents

• Models
• Setup
• Preparation
• Train
• Inference
• Experiments
• Citation

🔓 Models

📷 Setup

Please follow the instructions below to install the required packages.

1. Clone this repository

https://github.com/ZNLP/Implicit-Cross-Lingual-Rewarding.git

2. Install Package

conda create -n ICR python=3.10 -y
conda activate ICR
cd Implicit-Cross-Lingual-Rewarding
pip install -r requirements.txt

💡 Preparation

bash scripts/run_pipeline.sh

📈 Train

Our training is mostly performed on LLaMA-Factory code base. Please refer to that repo for more details.

📌 Inference

bash scripts/batch_inference_for_xalpacaeval.sh

👀 Experiments

We provide some results in this section. More detailed results can be found in our paper.

General Instruction Following

• X-alpacaEval Leaderboard

📊 More Analysis

• Different Implicit Rewards

🔗 Citation

If you find this repo useful for your research, please consider citing the paper

Implicit Cross-lingual Rewarding:

@article{yang2025implicit,
  title={Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment},
  author={Yang, Wen and Wu, Junhong and Wang, Chen and Zong, Chengqing and Zhang, Jiajun},
  journal={arXiv preprint arXiv:2503.04647},
  year={2025}
}

Language Imbalance Driven Rewarding:

@article{yang2024language,
  title={Language Imbalance Driven Rewarding for Multilingual Self-improving},
  author={Yang, Wen and Wu, Junhong and Wang, Chen and Zong, Chengqing and Zhang, Jiajun},
  journal={arXiv preprint arXiv:2410.08964},
  year={2024}
}

🌈 Acknowledgement

We would like to thank the following repos for their great work:

• LLaMA-Factory for providing the preference optimization framework
• Vllm for the efficient inference engine with high throughput
• transformers for providing the model-base and fune-tuning framework
• SimPO for providing Llama-3-Base-8B-SFT-DPO and Llama-3-Base-8B-SFT as base model in our work
• LLaMA for making Llama series models available

🔎 License

This project is released under the Apache 2.0 license. Parts of this project contain code and models from other sources, which are subject to their respective licenses.