Modelling Latent Translations for Cross-Lingual Transfer

Documentation | Cite | Paper

While often achieving SOTA results, “translation test” suffers from some limitations: 1) errors accumulate along the pipeline and cannot be corrected; 2) only the maximum-likelihood translation is generated, which is not necessarily the most relevant to the downstream task.

Instead, we integrate both translator and classifier into a single model, by treating the intermediate translations as a latent random variable. By initialising both components with pre-trained models, our method is suitable for few-shot learning.

By performing inference under this model, 1) we fine-tune the translator end-to-end or via Minimum Risk Training according to the downstream task loss; 2) we draw multiple translation samples to perform ensemble prediction in the downstream task.

For instance, the following example from Turkish XCOPA reveals how the maximum-likelihood translation leads to a wrong prediction. On the other hand, majority voting from multiple translation samples recover the correct prediction.

Prerequisites

• Python 3.7
• pip install -r requirements.txt

Data

The datasets for the experiments reported in the paper can be obtained with this command:

cd scripts
bash download_data.sh

The traslations into English of their test sets via Google MT can be found at the following links:

Experiments

For cross-lingual transfer based on neural machine translation:

python train.py --task_name <xcopa|pawsx|xnli> --mode <trans-soft|trans-hard> --nmt_model_name <marian|google|mbart50> --do_train --do_eval --do_refine --num_samples 12 --per_gpu_train_batch_size 1 --gradient_accumulation_steps 24 --learning_rate 8e-6 --per_gpu_eval_batch_size 2

Note that the number of translation samples is specified by --num_samples. By setting --mode trans-soft the translator parameters are fine-tuned with Minimum Risk Training, whereas --mode trans-hard keeps them frozen.

For cross-lingual transfer based on multilingual encoders:

python train.py --task_name <xcopa|pawsx|xnli> --mode multi --cls_model_name <xlmr|mbart50> --do_train --do_eval --do_refine --per_gpu_train_batch_size 4 --gradient_accumulation_steps 6 --learning_rate 8e-6 --max_seq_length 384 --num_train_epochs 2 --save_steps 1000

Visualisations

Our method yields larger gains for languages whose BLEU scores are lower. This plot shows the correlation between NMT BLEU score and gains compared to vanilla translate test.

Cite

@misc{ponti2021modelling,
      title={Modelling Latent Translations for Cross-Lingual Transfer}, 
      author={Edoardo Maria Ponti and Julia Kreutzer and Ivan Vulić and Siva Reddy},
      year={2021},
      eprint={2107.11353},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/pdf/2107.11353.pdf},
}