Multilingual Pre-training with Language and Task Adaptation for Multilingual Text Style Transfer (ACL 2022)

Skoltech NNLP educational project. This repository is not an original official implementation of the work, but a refactored codebase based on the code from https://github.com/laihuiyuan/multilingual-tst.git

Performed within the NNLP coursework at Skoltech.

Overview

In view of the general scarcity of parallel data, authors proposed a modular approach for multilingual formality transfer, which consists of two training strategies that target adaptation to both language and task. The approach achieves competitive performance without monolingual task-specific parallel data and can be applied to other style transfer tasks as well as to other languages.

Dataset

• News-crawl: Language-specific generic non-parallel data.
• RuDetox: Parallel Russian dataset for detoxification task
• EnDetox: Parallel English dataset for detoxification task

Quick Start

To train model you need to put data from News-crawl in Russian or English to data/news-crawl folder with name train.ru_RU, valid.ru_RU (train.en_XX, valid.en_XX). As train data we use news crawl for 2021 year, as valid data 50k lines from 2020 news crawl for each language.

Step 1: Language Adaptation Training

Non-parallel data from news-crawl required

# en_XX, ru_RU
python train_lang_adap.py -dataset news-crawl -lang en_XX

Step 2: Task Adaptation Training

Parallel data from data/detox required -- it's already here

# en_XX, ru_RU
python train_task_adap.py -dataset detox -lang en_XX

Step 3: Inference

# ADAPT + EN data (it_IT, fr_XX, pt_XX)
python infer_en_data.py -dataset detox -lang it_IT -style 0 

# ADAPT + EN cross-attn (it_IT, fr_XX, pt_XX)
python infer_en_attn.py -dataset detox -lang it_IT -style 0    

You can also reproduce results via this notebook

Citation

@inproceedings{lai-etal-2022-multi,
    title = "Multilingual pre-training with Language and Task Adaptation for Multilingual Text Style Transfer",
    author = "Lai, Huiyuan  and
      Toral, Antonio  and
      Nissim, Malvina",
    booktitle = "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics",
    month = May,
    year = "2022",
    publisher = "Association for Computational Linguistics",
}