aacl23-mnmt-tutorial

📘 slides | ▶️ Recording

Reading List

Fundamental concepts

Architecture

• Sequence to Sequence Learning with Neural Networks
  Paper

• Neural Machine Translation by Jointly Learning to Align and Translate
  Paper

• Attention Is All You Need
  Paper

• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
  Paper

• BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  Paper Code

Vocabulary

• Neural Machine Translation of Rare Words with Subword Units
  Paper Code

• Neural Machine Translation with Byte-Level Subwords
  Paper

• Neural Machine Translation with Byte-Level Subwords
  Paper

• SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing
  Paper Code

Prominent Massively Multilingual NMT systems

• Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation
  Paper

• Massively Multilingual Neural Machine Translation
  Paper

• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
  Paper

• Beyond English-Centric Multilingual Machine Translation (M2M-100)
  Paper Code

• Multilingual Denoising Pre-training for Neural Machine Translation (MBART-25)
  Paper Code

• Multilingual Translation from Denoising Pre-Training (MBART-50)
  Paper Code

• DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders
  Paper Code

• No Language Left Behind: Scaling Human-Centered Machine Translation (NLLB-200)
  Paper Code

• MADLAD-400: A Multilingual And Document-Level Large Audited Dataset
  Paper Model

• Towards the Next 1000 Languages in Multilingual Machine Translation: Exploring the Synergy Between Supervised and Self-Supervised Learning
  Paper

Models for related languages.

African

• MMTAfrica: Multilingual Machine Translation for African Languages
  Paper Code

• AfroMT: Pretraining Strategies and Reproducible Benchmarks for Translation of 8 African Languages
  Paper Code

• ANVITA-African: A Multilingual Neural Machine Translation System for African Languages
  Paper

• AfroLM: A Self-Active Learning-based Multilingual Pretrained Language Model for 23 African Languages
  Paper Code

Middle-East / North-African

• AraBERT: Transformer-based Model for Arabic Language Understanding
  Paper Code

• The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models (CAMeLBERT)
  Paper Code

South-East Asian

• SG Translate Together - Uplifting Singapore’s translation standards with the community through technology
  Paper

• IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation (IndoBART, IndoGPT)
  Paper Code

• WangchanBERTa: Pretraining transformer-based Thai Language Models
  Paper Code

• WangchanBERTa: Pretraining transformer-based Thai Language Models
  Paper Code

European languages

• OPUS-MT – Building open translation services for the World
  Paper Code

Indigenous languages of America

• IndT5: A Text-to-Text Transformer for 10 Indigenous Languages
  Paper Code

• Enhancing Translation for Indigenous Languages: Experiments with Multilingual Models
  Paper

Indian subcontinent

• Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages (IndicTrans1)
  Paper Code

• IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages
  Paper Code

• IndicBART: A Pre-trained Model for Indic Natural Language Generation
  Paper Code

China

• ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information
  Paper Code

• CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation
  Paper Code

Creoles

• KreolMorisienMT: A Dataset for Mauritian Creole Machine Translation
  Paper Code

• CreoleVal: Multilingual Multitask Benchmarks for Creoles
  Paper Code

Dataset Curation

Monolingual Data Curation - Large scale

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Data:C4, Model:T5)
  Paper Code

• mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (Data:mC4, Model:mT5)
  Paper Code

• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
  Paper Data

• The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only
  Paper

Monolingual Data Curation - Language-family specific

---

NOTE

We refer the reader to the papers on language-family specific models, as these include monolingual data creation, bitext mining and model training.

Additional papers other than those mentioned above are included in this subsection.

---

• IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages
  Paper Code

• Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages
  Paper Code

• Varta: A Large-Scale Headline-Generation Dataset for Indic Languages
  Paper Code

• WebCrawl African : A Multilingual Parallel Corpora for African Languages
  Paper Code

Parallel Corpora Creation

• CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs
  Paper Data

• Billion-scale similarity search with GPUs (FAISS)
  Paper Code

• CCMatrix: Mining Billions of High-Quality Parallel Sentences on the Web
  Paper Code

• xSIM++: An Improved Proxy to Bitext Mining Performance for Low-Resource Languages
  Paper Data

Sentence Embedding Models

• Language-agnostic BERT Sentence Embedding
  Paper Code

• LEALLA: Learning Lightweight Language-agnostic Sentence Embeddings with Knowledge Distillation
  Paper Code

• Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
  (LASER1)
  Paper Code

• Bitext Mining Using Distilled Sentence Representations for Low-Resource Languages
  (LASER3)
  Paper Code

• Multilingual Representation Distillation with Contrastive Learning (LASER3-CO)
  Paper

• Learning Multilingual Sentence Representations with Cross-lingual Consistency Regularization (MuSR)
  Paper Code

• SONAR: Sentence-Level Multimodal and Language-Agnostic Representations
  Paper Code

Data Quality v/s Scale

• Data and Parameter Scaling Laws for Neural Machine Translation
  Paper Code

• Data Scaling Laws in NMT: The Effect of Noise and Architecture
  Paper

• “A Little is Enough”: Few-Shot Quality Estimation based Corpus Filtering improves Machine Translation
  Paper

Human-annotated Seed Corpora

• The TDIL Program and the Indian Language Corpora Intitiative (ILCI)
  Paper

• Small Data, Big Impact: Leveraging Minimal Data for Effective Machine Translation
  Paper Data

• MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages
  Paper Data

Benchmarks

• The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali–English and Sinhala–English
  Paper

• The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation
  Paper Data

• NTREX-128 – News Test References for MT Evaluation of 128 Languages
  Paper Data

Modeling

---

NOTE

We refer the reader to the papers on massively multilingual models, as these include some aspects of modeling.

Additional papers other than those mentioned above are included in this subsection.

---

Vocabulary

• How Robust is Neural Machine Translation to Language Imbalance in Multilingual Tokenizer Training?
  Paper

• Out-of-the-box Universal Romanization Tool uroman
  Paper

• The IndicNLP Library
  Paper Code

• Pre-training via Leveraging Assisting Languages for Neural Machine Translation
  Paper Code

• BPE-Dropout: Simple and Effective Subword Regularization
  Paper

• Efficient Neural Machine Translation for Low-Resource Languages via Exploiting Related Languages
  Paper Code

• Exploiting Language Relatedness for Low Web-Resource Language Model Adaptation: An Indic Languages Study
  Paper Code

• Language Relatedness and Lexical Closeness can help Improve Multilingual NMT: IITBombay@MultiIndicNMT WAT2021
  Paper

• Auxiliary Subword Segmentations as Related Languages for Low Resource Multilingual Translation
  Paper

• Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languages
  Paper Code

• Transfer Learning in Multilingual Neural Machine Translation with Dynamic Vocabulary
  Paper

Leveraging Ordering Information

• Addressing word-order Divergence in Multilingual Neural Machine Translation for extremely Low Resource Languages Paper Code

• Language Related Issues for Machine Translation between Closely Related South Slavic Languages
  Paper

• A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space
  Paper Code

• Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Review
  Paper

• Decomposed Prompting for Machine Translation Between Related Languages using Large Language Models
  Paper Code

• JASS: Japanese-specific Sequence to Sequence Pre-training for Neural Machine Translation
  Paper Code

Training

Joint Training / Language-Relatedness

• Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism
  Paper

• Multi-Task Learning for Multiple Language Translation
  Paper

• Multi-Task Learning for Multiple Language Translation
  Paper

• Contact Relatedness can help improve multilingual NMT: Microsoft STCI-MT @ WMT20
  Paper

• Investigating Multilingual NMT Representations at Scale
  Paper

• Enabling Multi-Source Neural Machine Translation By Concatenating Source Sentences In Multiple Languages
  Paper

• Multilingual Neural Machine Translation with Language Clustering
  Paper

• Bridging Linguistic Typology and Multilingual Machine Translation with Multi-View Language Representations
  Paper Code

• Delexicalized Cross-lingual Dependency Parsing for Xibe
  Paper

• An Empirical Study of Language Relatedness for Transfer Learning in Neural Machine Translation
  Paper

• Efficient Unsupervised NMT for Related Languages with Cross-Lingual Language Models and Fidelity Objectives
  Paper

• Adapting High-resource NMT Models to Translate Low-resource Related Languages without Parallel Data
  Paper

• Linguistically-driven Multi-task Pre-training for Low-resource Neural Machine Translation
  Paper Code

Data Curriculum / Multi-stage training

• Instance Weighting for Neural Machine Translation Domain Adaptation
  Paper Code

• Exploiting Multilingualism through Multistage Fine-Tuning for Low-Resource Neural Machine Translation
  Paper

• Data Selection Curriculum for Neural Machine Translation
  Paper

Modeling

Mixture of Experts

• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
  Paper

• ST-MoE: Designing Stable and Transferable Sparse Expert Models
  Paper Code

• Towards Understanding Mixture of Experts in Deep Learning
  Paper Code

• Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference
  Paper

• Universal Neural Machine Translation for Extremely Low Resource Languages
  Paper Code

• Transfer Learning across Low-Resource, Related Languages for Neural Machine Translation
  Paper Code

Decoder-only MT models

• Examining Scaling and Transfer of Language Model Architectures for Machine Translation (LM4MT)
  Paper

• ALMA: Advanced Language Model-based translator
  Paper Code

Zero-shot transfer-learning / Adaptation to new languages.

• Rapid Adaptation of Neural Machine Translation to New Languages
  Paper Code

• Improving Zero-Shot Cross-lingual Transfer Between Closely Related Languages by Injecting Character-Level Noise
  Paper

• Utilizing Lexical Similarity to Enable Zero-Shot Machine Translation for Extremely Low-resource Languages
  Paper

• Improving Zero-Shot Translation by Disentangling Positional Information
  Paper Code

• Simple, Scalable Adaptation for Neural Machine Translation Paper

• T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translation
  Paper

• Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
  Paper Code

• From Bilingual to Multilingual Neural Machine Translation by Incremental Training
  Paper

• Language-Family Adapters for Low-Resource Multilingual Neural Machine Translation
  Paper

• Improving Neural Machine Translation of Indigenous Languages with Multilingual Transfer Learning
  Paper

Model Compression

• Sequence-Level Knowledge Distillation
  Paper Code

• Learning both Weights and Connections for Efficient Neural Networks
  Paper

• Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Model
  Paper Code

• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
  Paper Code

• The case for 4-bit precision: k-bit Inference Scaling Laws Paper

• An Empirical Study of Leveraging Knowledge Distillation for Compressing Multilingual Neural Machine Translation Models
  Paper Code

• Multilingual Neural Machine Translation with Language Clustering
  Paper

Evaluation

Automatic Evaluation

• Bleu: a Method for Automatic Evaluation of Machine Translation
  Paper

• chrF: character n-gram F-score for automatic MT evaluation
  Paper

• chrF++: words helping character n-grams
  Paper

• A Call for Clarity in Reporting BLEU Scores
  Paper Code

• BLEURT: Learning Robust Metrics for Text Generation
  Paper Code

• Learning Compact Metrics for MT
  Paper

• IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation Metrics for Indian Languages
  Paper Code

• COMET: A Neural Framework for MT Evaluation
  Paper Code

• Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMET
  Paper Code

• Extrinsic Evaluation of Machine Translation Metrics
  Paper

• Large Language Models Are State-of-the-Art Evaluators of Translation Quality
  Paper Code

• The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation Paper

Human Evaluation

• Continuous Measurement Scales in Human Evaluation of Machine Translation
  Paper

• Is Machine Translation Getting Better over Time?
  Paper

• Multidimensional quality metrics: a flexible system for assessing translation quality
  Paper

• Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation
  Paper Code

• SemEval-2016 Task 1: Semantic Textual Similarity, Monolingual and Cross-Lingual Evaluation
  Paper

• Consistent Human Evaluation of Machine Translation across Language Pairs
  Paper

Toolkits

• FairseqV1
• FairseqV2
• Transformers
• tensor2tensor
• trax
• OpenNMT
• MarianNMT
• JoeyNMT
• Sockeye
• YANMTT

Citation

@InProceedings{gala-chitale-dabre:2023:ijcnlp,
  author    = {Gala, Jay  and  Chitale, Pranjal A.  and  Dabre, Raj},
  title     = {Developing State-Of-The-Art Massively Multilingual Machine Translation Systems for Related Languages},
  booktitle      = {Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics},
  month          = {November},
  year           = {2023},
  address        = {Nusa Dua, Bali},
  publisher      = {Association for Computational Linguistics},
  pages     = {35--42},
  url       = {https://aclanthology.org/2023.ijcnlp-tutorials.6}
}