The Optimizer, Not the Reward: Negative Signals Drive Multilingual Alignment

This repository reproduces the experiments in "The optimizer, not the reward, drives cross-lingual transfer in multilingual alignment". We cross four optimization objectives with three multilingual preference estimators (LIDR, ICR, LaCoMSA) and two non-learned controls (random and heuristic pairing), holding training data and architecture fixed.

Key finding: Negative-weighted optimization (W-REINFORCE) consistently outperforms DPO and positive-only baselines, and its gains persist even with random or simple heuristic pairing — suggesting that active negative reinforcement, not reward sophistication, is the primary driver of cross-lingual transfer.

Figure 1. (A) Prior multilingual alignment work attributes gains to cross-lingual reward construction; this work attributes them to the negative component of the optimizer. (B) Average win rate across 5 languages (ICR and LaCoMSA estimators). Negative-weighted W-REINFORCE stays above baseline under learned, heuristic, and random pairing, while positive-only (SFT, PPO) and balanced (DPO) objectives remain at or below it.

Figure 2. Experimental framework for isolating the role of the optimization objective. Left (Preference Pair Generation): A base model generates multilingual candidate responses, scored by cross-lingual reward functions spanning sophisticated learned rewards (LIDR, ICR, LaCoMSA) and simple heuristic rewards (random, simple heuristic), producing one preference dataset per reward. Right (Multilingual Alignment): Each preference dataset is split into positive (chosen) and negative (rejected) sets and used to train five objectives that vary how strongly they leverage the negative signal: positive-only (SFT, PPO), balanced (DPO), negative-weighted (W-REINFORCE, marked with a star), and negative-only (NPO).

---

Overview

---

Repository Layout

.
├── configs/
│   ├── model_config.yaml          # HuggingFace model keys (Llama3, Qwen2.5)
│   ├── methods_sources.yaml       # External method paths & envs
│   ├── pairs/                     # Pair-generation configs (experiment / test variants)
│   └── train/                     # Baseline DPO configs (LIDR, ICR, LaCoMSA)
├── external/
│   ├── icr/                       # Implicit Cross-Lingual Reward (ICR)
│   ├── lacomsa/                   # Language-Consistency Multilingual Self-Alignment (LaCoMSA)
│   └── lidr/                      # Language-Imbalance-Driven Rewarding (LIDR)
├── data/
│   ├── pairs/standard/            # Standardized preference pairs by method & EXP_ID
│   └── raw/                       # Raw UltraFeedback data
├── models/
│   ├── rl/<method>/<algo>/<EXP_ID>/   # RL-trained models
│   └── sft/<method>/<EXP_ID>/         # SFT-finetuned checkpoints
├── scripts/
│   ├── pairs/                     # Pair generation scripts
│   ├── setup_env/                 # One-env-per-method setup scripts
│   └── train/                     # Training entrypoint & scripts
└── src/                           # Core utilities (generation, evaluation, helpers)

---

Environment Setup

One conda environment per method. All environments live in the same conda installation; activate the one you need before running.

# ICR
bash scripts/setup_env/setup_icr.sh
conda activate rail-icr

# LIDR
bash scripts/setup_env/setup_lidr.sh
conda activate rail-lidr

# LaCoMSA
bash scripts/setup_env/setup_lacomsa.sh
conda activate rail-lacomsa

# RL training (NPO / PPO / W-REINFORCE / SFT)
bash scripts/setup_env/setup_train.sh --fresh
conda activate rail-train

# Baseline DPO (TRL-based ICR only)
bash scripts/setup_env/setup_icr_dpo/setup.sh
conda activate rail-icr-dpo

Note: On machines without a CUDA toolchain, skip deepspeed/apex in the rail-train env. On GPU servers, install DeepSpeed and pass --deepspeed to the training script.

---

Pair Generation

All pair-generation scripts read PAIRS_CONFIG and write to data/pairs/standard/<method>/<EXP_ID>/train.jsonl. Choose the experiment config (ID-only, 2 languages) or the test config (local smoke test).

# Choose an EXP_ID once; reuse it across all stages
export EXP_ID="exp01_id_enesru_llama3_8b"

# --- LIDR ---
conda activate rail-lidr
EXP_ID="$EXP_ID" PAIRS_CONFIG="configs/pairs/lidr_experiment.json" \
  bash scripts/pairs/lidr/generate_pairs.sh

# --- ICR ---
conda activate rail-icr
EXP_ID="$EXP_ID" PAIRS_CONFIG="configs/pairs/icr_experiment.json" \
  bash scripts/pairs/icr/generate_pairs.sh

# --- LaCoMSA ---
conda activate rail-lacomsa
EXP_ID="$EXP_ID" PAIRS_CONFIG="configs/pairs/lacomsa_experiment.json" \
  bash scripts/pairs/lacomsa/generate_pairs.sh

Output: data/pairs/standard/<method>/<EXP_ID>/train.jsonl

---

Training

Configuration

Edit configs/train.yaml (or use an alternate file via TRAIN_YAML):

Loss coefficients in configs/train.yaml:

po:
  npo_coeff: 1.0    # NPO (rejected-only) weight
  ppo_coeff: 0.1    # PPO (chosen-only) weight  ← λ in the paper
  beta: 0.1         # DPO temperature

Run Training

conda activate rail-train

# Finetune (SFT) first — run once per method
TRAIN_YAML=train.yaml METHOD=icr EXP_ID="$EXP_ID" \
  bash scripts/train/post_training.sh --only-finetune

# Then run PO — one job per objective
TRAIN_YAML=train.yaml METHOD=icr PO_LOSS=npo EXP_ID="$EXP_ID" \
  bash scripts/train/post_training.sh --only-train

TRAIN_YAML=train.yaml METHOD=icr PO_LOSS=ppo EXP_ID="$EXP_ID" \
  bash scripts/train/post_training.sh --only-train

TRAIN_YAML=train.yaml METHOD=icr PO_LOSS=w-reinforce EXP_ID="$EXP_ID" \
  bash scripts/train/post_training.sh --only-train

Repeat for METHOD=lidr and METHOD=lacomsa.

With DeepSpeed (GPU servers with DeepSpeed installed):

METHOD=icr PO_LOSS=w-reinforce EXP_ID="$EXP_ID" \
  bash scripts/train/post_training.sh --only-train --deepspeed

Baseline DPO (full pipeline)

# LIDR
EXP_ID="$EXP_ID" \
PAIRS_CONFIG="configs/pairs/lidr_experiment.json" \
DPO_CONFIG="configs/train/lidr/dpo_base.json" \
bash scripts/train/baselines/lidr/lidr_fullpipeline_run.sh

# ICR (TRL DPO)
EXP_ID="$EXP_ID" \
PAIRS_CONFIG="configs/pairs/icr_experiment.json" \
DPO_CONFIG="configs/train/icr/dpo_trl_base.json" \
bash scripts/train/baselines/icr/icr_fullpipeline_run.sh

# LaCoMSA
EXP_ID="$EXP_ID" \
PAIRS_CONFIG="configs/pairs/lacomsa_experiment.json" \
DPO_CONFIG="configs/train/lacomsa/dpo_base.json" \
bash scripts/train/baselines/lacomsa/lacomsa_fullpipeline_run.sh

---

Logging (Weights & Biases)

wandb login
# or set: export WANDB_API_KEY=<your-key>

# If your org disables personal entities:
export WANDB_ENTITY=<your-team-slug>

Metrics are reported under project NeMA. WANDB_MODE=offline keeps runs local.

---

Output Locations