Domain Adaptation for Neural Machine Translation in Low-Resource Enivronments

This repository contains a comprehensive pipeline for neural machine translation research, specifically focusing on Russian-English translation using various backtranslation techniques, data augmentation methods, and quality assessment tools. The pipeline supports both traditional machine translation models and large language models (LLMs) for backtranslation generation.

Overview

The research pipeline is designed to:

1. Generate synthetic parallel data through multiple backtranslation approaches
2. Evaluate translation quality using multiple metrics and diagnostic tools
3. Train and fine-tune translation models with various datasets
4. Filter and clean datasets using quality-based filtering
5. Perform domain classification to distinguish medical from non-medical text

📝 Note: Each script in this repository contains detailed documentation and instructions within the file itself. Please refer to the docstrings and comments in each script for additional implementation details, parameter explanations, and usage examples.

Table of Contents

• Installation
• Dataset Management
• Available Datasets
• Dataset Filtering
• Data Augmentation
• Backtranslation Methods
• Quality Evaluation
• Model Training
• Domain Classification
• Diagnostic Analysis
• File Reference
• Workflow Examples

Installation

Dependencies

pip install -r requirements.txt

Hugging Face Setup

Set your Hugging Face token in each script or as an environment variable:

export HF_TOKEN='your_huggingface_token_here'

Dataset Management

Dataset Size Calculation

File: dataset_size.py

Calculate the number of examples in various datasets to understand data availability.

python dataset_size.py

Purpose:

• Monitors dataset sizes across multiple backtranslation variants
• Tracks filtered, corrupted, and augmented dataset versions
• Helps with experimental planning and resource allocation

Outputs: Console display of dataset sizes in tabular format

Available Datasets

This section lists all the datasets available in the pipeline, organized by category. These datasets can be used for training, evaluation, and analysis across all pipeline components.

Base Parallel Datasets

Original Human-Translated Datasets:

• sethjsa/medline_en_ru_parallel - Medical parallel corpus (Medline)
• sethjsa/scipar_en_ru_parallel - Scientific paper abstracts parallel corpus

Traditional Machine Translation Backtranslation

Basic Backtranslation Datasets:

• giannhskp/scipar_ru_en_backtranslation - SciPar RU→EN backtranslation
• giannhskp/medline-parallel_ru_en_backtranslation - Medline parallel RU→EN backtranslation
• giannhskp/medline_ru_en_backtranslation - Medline RU→EN backtranslation

LLM-Generated Backtranslation Datasets

Core LLM Backtranslations:

• giannhskp/medline_ru_en_llm-backtranslation - LLM-generated Medline backtranslation
• giannhskp/medline-parallel_ru_en_llm-backtranslation - LLM-generated Medline parallel backtranslation
• giannhskp/scipar_ru_en_llm-backtranslation - LLM-generated SciPar backtranslation

LLM Backtranslation Variants (Different Decoding Strategies):

• giannhskp/medline_ru_en_llm-backtranslation_beam5 - Beam search (5 beams)
• giannhskp/medline-parallel_ru_en_llm-backtranslation_beam5 - Beam search (5 beams)
• giannhskp/scipar_ru_en_llm-backtranslation_beam5 - Beam search (5 beams)
• giannhskp/medline_ru_en_llm-backtranslation_sampling - Sampling (top_p=0.9, temp=0.9)
• giannhskp/medline-parallel_ru_en_llm-backtranslation_sampling - Sampling strategy
• giannhskp/scipar_ru_en_llm-backtranslation_sampling - Sampling strategy

Quality-Filtered Datasets

Score-Based Filtering (40.0 threshold):

• giannhskp/medline_en_ru_parallel_filtered_40.0
• giannhskp/medline_ru_en_backtranslation_filtered_40.0
• giannhskp/medline-parallel_ru_en_backtranslation_filtered_40.0
• giannhskp/scipar_ru_en_backtranslation_filtered_40.0
• giannhskp/scipar_en_ru_parallel_filtered_40.0

CometKiwi-Based Filtering (60.0 threshold):

• giannhskp/medline_en_ru_parallel_filtered_60.0_wmt22-cometkiwi-da
• giannhskp/medline-parallel_ru_en_backtranslation_filtered_60.0_wmt22-cometkiwi-da
• giannhskp/medline_ru_en_backtranslation_filtered_60.0_wmt22-cometkiwi-da
• giannhskp/scipar_ru_en_backtranslation_filtered_60.0_wmt22-cometkiwi-da
• giannhskp/scipar_en_ru_parallel_filtered_60.0_wmt22-cometkiwi-da

LLM Backtranslation + CometKiwi Filtering (40.0 threshold):

• giannhskp/medline-parallel_ru_en_llm-backtranslation_filtered_40.0_wmt22-cometkiwi-da
• giannhskp/medline_ru_en_llm-backtranslation_filtered_40.0_wmt22-cometkiwi-da
• giannhskp/scipar_ru_en_llm-backtranslation_filtered_40.0_wmt22-cometkiwi-da

Domain-Classifier Filtered Datasets

Medical Domain Classification Filtering:

• giannhskp/classifier-filtered-medline_en_ru_parallel
• giannhskp/classifier-filtered-medline-parallel_ru_en_backtranslation
• giannhskp/classifier-filtered-scipar_en_ru_parallel
• giannhskp/classifier-filtered-scipar_ru_en_backtranslation
• giannhskp/classifier-filtered-medline_ru_en_backtranslation
• giannhskp/classifier-filtered-scipar_ru_en_llm-backtranslation
• giannhskp/classifier-filtered-scipar_ru_en_llm-backtranslation_filtered_40.0_wmt22-cometkiwi-da

Length-Filtered Datasets

Minimum Length Filtering (>5 words):

• giannhskp/medline-parallel_ru_en_llm-backtranslation_filtered_40.0_wmt22-cometkiwi-da_gt5
• giannhskp/medline_ru_en_llm-backtranslation_filtered_40.0_wmt22-cometkiwi-da_gt5
• giannhskp/medline_en_ru_parallel_gt5
• giannhskp/classifier-filtered-scipar_ru_en_llm-backtranslation_filtered_40.0_wmt22-cometkiwi-da_gt5

Corrupted/Augmented Datasets

Word Deletion Augmentation (various rates):

• giannhskp/medline_ru_en_llm-backtranslation_filtered_40.0_wmt22-cometkiwi-da_word_deletion_0.33_0.15
• giannhskp/medline_ru_en_llm-backtranslation_filtered_40.0_wmt22-cometkiwi-da_wd_0.5_0.15
• giannhskp/medline_ru_en_llm-backtranslation_filtered_40.0_wmt22-cometkiwi-da_wd_0.33_0.33
• giannhskp/medline_ru_en_llm-backtranslation_filtered_40.0_wmt22-cometkiwi-da_wd_1_0.15
• giannhskp/classifier-scipar_ru_en_llm-backtranslation_fi_40.0_wmt22-cometkiwi-da_word_deletion_0.33_0.15
• giannhskp/classifier-scipar_ru_en_llm-backtranslation_filtered_40.0_wmt22-cometkiwi-da_wd_0.5_0.15
• giannhskp/classifier-scipar_ru_en_llm-backtranslation_filtered_40.0_wmt22-cometkiwi-da_wd_0.33_0.33
• giannhskp/classifier-filtered-scipar_ru_en_llm-backtranslation_filtered_40.0_wmt22-cometkiwi-da_wd_1_0.15
• giannhskp/medline-parallel_ru_en_llm-backtranslation_filtered_40.0_wmt22-cometkiwi-da_wd_0.33_0.15
• giannhskp/medline-parallel_ru_en_llm-backtranslation_filtered_40.0_wmt22-cometkiwi-da_wd_0.5_0.15
• giannhskp/medline-parallel_ru_en_llm-backtranslation_filtered_40.0_wmt22-cometkiwi-da_wd_0.33_0.33
• giannhskp/medline-parallel_ru_en_llm-backtranslation_filtered_40.0_wmt22-cometkiwi-da_wd_1_0.15

Dataset Filtering

The pipeline provides three advanced filtering techniques to improve dataset quality by removing low-quality translations and out-of-domain content.

1. COMET-Based Quality Filtering

File: custom_utils/filter.py

Neural-based translation quality filtering using COMET (Crosslingual Optimized Metric for Evaluation of Translation) models.

python custom_utils/filter.py \
    --input_dataset "giannhskp/medline-parallel_ru_en_llm-backtranslation" \
    --src_field "en" \
    --mt_field "ru" \
    --hf_username "your_username" \
    --threshold 0.4 \
    --use_gpu

Key Features:

• Neural Quality Assessment: More accurate than traditional metrics like BLEU
• Reference-Free Evaluation: Works without gold standard translations
• Flexible Thresholds: Configurable quality score cutoffs
• Batch Processing: GPU-accelerated scoring for large datasets

Supported COMET Models:

• Unbabel/wmt22-cometkiwi-da (default) - CometKiwi reference-free model
• Unbabel/wmt20-comet-qe-da - Quality estimation model

Command Line Options:

• --input_dataset: HuggingFace dataset name to filter
• --threshold: COMET score threshold (default: 0.6)
• --src_field/--mt_field: Column names for source and translation text
• --comet_model: Specific COMET model to use
• --hf_username: Username for uploading filtered dataset
• --use_gpu: Enable GPU acceleration

Output Format: Filtered datasets are uploaded to Hub with naming convention: {username}/{dataset_name}_filtered_{threshold*100}_{model_name}

Example: myuser/translation-data_filtered_40_wmt22-cometkiwi-da

2. Domain Classification Filtering

File: custom_utils/filter_with_classifier.py

Filters datasets using pre-trained text classification models to keep only domain-relevant content (e.g., medical vs. non-medical text).

python custom_utils/filter_with_classifier.py \
    --dataset_names giannhskp/scipar_ru_en_llm-backtranslation \
    --model_name_or_path "giannhskp/xlm-roberta-base_domain_classifier" \
    --text_column "en" \
    --label_to_keep 1 \
    --push_to_hub \
    --hub_repo_prefix "giannhskp/classifier-filtered-"

Key Features:

• Domain-Specific Filtering: Uses multilingual classifiers for content filtering
• Batch Processing: Efficient inference with configurable batch sizes
• Multi-Split Support: Processes train/validation/test splits automatically
• Flexible Integration: Works with any text classification model

Typical Workflow:

1. Train domain classifier using domain_classifier_train.py
2. Apply classifier to filter synthetic datasets
3. Keep only high-confidence domain-relevant examples

Command Line Options:

• --dataset_names: One or more datasets to filter
• --model_name_or_path: Pre-trained classifier model path/name
• --text_column: Column containing text to classify
• --label_to_keep: Target label to retain (1=medical, 0=non-medical)
• --batch_size: Inference batch size (default: 32)
• --push_to_hub: Upload filtered datasets to Hub
• --hub_repo_prefix: Repository naming prefix

Integration with Domain Classifier:

# 1. Train multilingual domain classifier
python domain_classifier_train.py --num_epochs 3

# 2. Filter datasets using trained classifier  
python custom_utils/filter_with_classifier.py \
    --dataset_names your_datasets \
    --model_name_or_path "username/xlm-roberta-base_domain_classifier" \
    --label_to_keep 1

3. Length-Based Filtering

File: dataset_notebooks/filter_by_length.ipynb

Interactive notebook for filtering translation datasets by minimum text length to remove low-quality or incomplete examples.

jupyter notebook dataset_notebooks/filter_by_length.ipynb

Purpose:

• Removes very short translation examples (< 5 characters) that may be low-quality
• Improves dataset quality for training translation models
• Simple interactive interface for dataset selection and processing

Key Features:

• Interactive Dataset Selection: Choose from predefined list of medical/scientific datasets
• Length-Based Filtering: Removes examples where Russian text is shorter than 5 characters
• Automatic Re-upload: Pushes filtered dataset back to Hub with "_gt5" suffix
• Multi-Format Support: Handles different dataset formats and splits automatically

Workflow:

1. Select Dataset: Interactive menu to choose from available datasets
2. Load Dataset: Automatically handles different dataset formats and splits
3. Apply Filter: Removes short Russian text examples (len(example['ru']) < 5)
4. Upload Result: Saves filtered dataset as {original_name}_gt5

Available Dataset Categories:

• Original parallel datasets (e.g., medline_en_ru_parallel)
• LLM-backtranslated datasets
• Pre-filtered datasets (COMET/classifier filtered)

Filter Criteria:

# Keeps only examples where Russian text has ≥ 5 characters
dataset.filter(lambda example: len(example['ru']) >= 5)

Output Naming: Filtered datasets are uploaded with "_gt5" suffix:

• Input: giannhskp/medline_ru_en_llm-backtranslation
• Output: giannhskp/medline_ru_en_llm-backtranslation_gt5

Integration with Other Filters: Length filtering is often applied as a final step after quality and domain filtering to ensure minimum content requirements.

4. Combined Filtering Strategies

Multi-Stage Filtering Pipeline:

1. Quality Filtering: Remove low-quality translations using COMET scores
2. Domain Filtering: Keep only domain-relevant content using classifiers
3. Length Filtering: Remove sentences below minimum character count. Reasoning: Removes incomplete or low-content examples

Example Multi-Stage Workflow:

# Stage 1: COMET quality filtering
python custom_utils/filter.py \
    --input_dataset "original_backtranslation_dataset" \
    --threshold 0.4 \
    --hf_username "username"

# Stage 2: Domain classification filtering  
python custom_utils/filter_with_classifier.py \
    --dataset_names "username/dataset_filtered_40_wmt22-cometkiwi-da" \
    --model_name_or_path "username/domain_classifier" \
    --label_to_keep 1 \
    --push_to_hub \
    --hub_repo_prefix "username/classifier-filtered-"

# Stage 3: Length filtering (using notebook interactively)
# jupyter notebook dataset_notebooks/filter_by_length.ipynb
# Select: username/classifier-filtered-dataset_filtered_40_wmt22-cometkiwi-da
# Result: username/classifier-filtered-dataset_filtered_40_wmt22-cometkiwi-da_gt5

Dataset Naming Conventions:

• Quality filtered: {dataset}_filtered_{threshold}_{comet_model}
• Domain filtered: classifier-filtered-{dataset}
• Length filtered: {dataset}_gt5
• Combined: classifier-filtered-{dataset}_filtered_{threshold}_{comet_model}_gt5

Data Augmentation

The pipeline includes data augmentation techniques to improve model robustness and create noisy training data for testing purposes.

Word Deletion Corruption

File: dataset_notebooks/corrupt_data.ipynb

Interactive tool for creating corrupted versions of translation datasets by randomly deleting words from English text.

jupyter notebook dataset_notebooks/corrupt_data.ipynb

Purpose:

• Robustness Testing: Creates noisy data to test model performance under degraded conditions
• Data Augmentation: Generates corrupted training examples for improved generalization
• Noise Simulation: Simulates real-world text corruption scenarios

Core Functionality:

• Word-Level Deletion: Randomly removes linguistic words from English sentences
• Configurable Probabilities: Separate controls for sentence selection and word deletion rates
• NLTK Tokenization: Uses linguistic word boundaries for natural corruption
• Proper Reconstruction: TreebankWordDetokenizer ensures grammatically sound output

Key Parameters:

SENTENCE_CORRUPTION_PROBABILITY = 1.0    # Probability a sentence gets corrupted
WORD_DELETION_PROBABILITY = 0.15         # Probability each word gets deleted

Workflow:

1. Setup: Download NLTK resources and initialize detokenizer
2. Selection: Choose target dataset from interactive list
3. Loading: Load dataset from Hugging Face Hub
4. Corruption: Apply word deletion with configured probabilities
5. Upload: Push corrupted dataset with systematic naming

Core Function - delete_linguistic_words_mapper(example):

• Tokenizes English text using NLTK word_tokenize
• Randomly deletes words based on WORD_DELETION_PROBABILITY
• Reconstructs sentences using TreebankWordDetokenizer
• Handles edge cases (empty strings, complete deletion)

Target Datasets: Focuses on 6 LLM-backtranslation datasets:

• giannhskp/medline_ru_en_llm-backtranslation
• giannhskp/scipar_ru_en_llm-backtranslation
• giannhskp/medline-parallel_ru_en_llm-backtranslation
• Plus their filtered variants (_filtered_40.0_wmt22-cometkiwi-da)

Example Usage:

# Configure corruption parameters
SENTENCE_CORRUPTION_PROBABILITY = 1.0
WORD_DELETION_PROBABILITY = 0.15

# Load and corrupt dataset
dataset = dataset_selector()  # Interactive selection
hf_dataset = load_hf_dataset(dataset)
corrupted_dataset = hf_dataset.map(delete_linguistic_words_mapper)

# Upload with automatic naming
repo_id = dataset + f"_wd_{SENTENCE_CORRUPTION_PROBABILITY}_{WORD_DELETION_PROBABILITY}"
corrupted_dataset.push_to_hub(repo_id, private=True)

Output Naming Convention: {original_name}_wd_{sentence_prob}_{word_prob}

Examples:

• Input: giannhskp/medline_ru_en_llm-backtranslation_filtered_40.0_wmt22-cometkiwi-da
• Output: giannhskp/medline_ru_en_llm-backtranslation_filtered_40.0_wmt22-cometkiwi-da_wd_1_0.15

Research Applications:

• Robustness Evaluation: Test model performance on degraded input
• Noise Tolerance Training: Train models to handle incomplete text
• Data Scarcity Scenarios: Augment limited training data with variations
• Error Analysis: Study model behavior under systematic corruption

🔧 Note: See the notebook's documentation for detailed parameter tuning guidelines.

Backtranslation Methods

The pipeline supports three main backtranslation approaches:

1. LLM-Based Backtranslation

File: llm-backtranslation-Tower.py

Uses the TowerInstruct-13B model for high-quality backtranslation from Russian to English.

python llm-backtranslation-Tower.py

Configuration Options:

• SOURCE_LANG: Source language (default: 'ru')
• TARGET_LANG: Target language (default: 'en')
• MODEL_NAME: LLM model to use (default: "Unbabel/TowerInstruct-13B-v0.1")
• DATASET_NAME: Input dataset containing Russian text
• BATCH_SIZE: Processing batch size (default: 1 for LLMs)
• PUSH_TO_HUB: Whether to upload results to Hugging Face Hub

Key Features:

• Configurable decoding strategies (beam search, sampling, greedy)
• Batch processing with progress tracking
• Automatic prompt formatting for translation tasks
• Memory-efficient processing for large datasets

Example Usage:

# Modify these variables in the script:
DATASET_NAME = "sethjsa/medline_en_ru_parallel"
OUTPUT_DATASET = "data/medline-parallel_ru_en_llm-backtranslation"
MODEL_NAME = "Unbabel/TowerInstruct-13B-v0.1"

2. Pivot-Based Backtranslation

File: pivot.py

Implements pivot translation using French as an intermediate language to generate English-Russian parallel data.

python pivot.py

Two Translation Pipelines:

1. Pivot-to-Source Translation:

   • French → English translation
   • Creates EN-RU pairs from existing FR-RU data

2. Target-to-Source via Pivot:

   • Russian → French → English translation
   • Double translation for additional synthetic data

Configuration:

• MODEL_SRC_PIVOT: French→English model (default: Helsinki-NLP OPUS)
• MODEL_PIVOT_TGT: Russian→French model (default: Helsinki-NLP OPUS)
• DATASET_SRC_PIVOT: Source datasets for FR-EN pairs
• DATASET_PIVOT_TGT: Source datasets for FR-RU pairs

Outputs:

• data/pivot_en-fr-ru_pivot_to_source_finetune
• data/pivot_en-fr-ru_target_to_source_finetune

3. Traditional Machine Translation Backtranslation

File: original_backtranslation.ipynb

Comprehensive notebook for creating traditional backtranslation datasets using neural machine translation models.

jupyter notebook original_backtranslation.ipynb

Two Main Workflows:

1. Monolingual-to-Backtranslation:

   • Converts monolingual text (e.g., Russian) into synthetic parallel data
   • Uses Helsinki-NLP Russian→English models for translation
   • Creates datasets like data/medline_ru_en_backtranslation

2. Parallel Dataset Expansion:

   • Creates multiple translation variants from existing parallel data
   • Generates N diverse translations per source sentence using sampling
   • Expands dataset size by factor of N (e.g., data/medline-parallel_ru_en_expanded-3)

Configuration Options:

cfg.SOURCE_LANG = "ru"  # Source language
cfg.TARGET_LANG = "en"  # Target language
cfg.MODEL_NAME_OR_PATH = "Helsinki-NLP/opus-mt-ru-en"
cfg.PER_DEVICE_EVAL_BATCH_SIZE = 256  # Translation batch size

Translation Quality Control:

• Deterministic Translation: Uses beam search (num_beams=15) for high-quality backtranslation
• Diverse Translation: Uses sampling (temperature=1.0, top_k=50, top_p=0.95) for dataset expansion

Available Source Datasets:

• sethjsa/medline_ru_mono - Russian medical abstracts (monolingual)
• sethjsa/medline_en_ru_parallel - Medical parallel data (for expansion)
• sethjsa/scipar_ru_mono - Russian scientific papers (monolingual)

Output Integration: Generated datasets can be loaded and used in training pipelines using load_from_disk() function.

📚 Note: For comprehensive step-by-step instructions, detailed configuration options, and complete code examples, refer to the extensive documentation within the original_backtranslation.ipynb notebook itself.

Quality Evaluation

1. Backtranslation Quality Assessment

File: evaluate-bt-quality.py

Comprehensive evaluation of LLM-generated backtranslations using multiple metrics.

python evaluate-bt-quality.py

Metrics Computed:

• CometKiwi: Reference-free quality estimation
• BLEU: n-gram overlap with reference translations
• COMET: Neural-based evaluation metric

Key Features:

• Batch processing for efficiency
• CSV output for detailed analysis
• Comparative evaluation across multiple datasets
• Summary statistics for quick assessment

Expected Results: The script includes commented results showing performance comparisons between different generation strategies (greedy, beam search, sampling).

2. LLM Translation Evaluation

File: evaluate-llm.py

Evaluates LLM translation performance on standard test sets.

python evaluate-llm.py

Test Datasets:

• WMT20 Biomedical (biomedical domain)
• FLORES (low-resource benchmark)
• TICO-19 (COVID-19 domain)

Features:

• Direct LLM evaluation without fine-tuning
• Multiple domain assessment
• Detailed per-dataset results
• CSV output for analysis

3. Fine-tuned Model Evaluation

File: evaluation.ipynb

Comprehensive evaluation framework for assessing fine-tuned translation models on multiple test datasets.

jupyter notebook evaluation.ipynb

Purpose:

• Evaluate your fine-tuned models after training with re-train.py or train.ipynb
• Compare model performance across different test sets
• Generate detailed evaluation reports and translations

Key Features:

• Multi-dataset evaluation on standard benchmarks (WMT, FLORES, TICO)
• Comprehensive metrics including BLEU, COMET, and other translation quality measures
• Batch inference with configurable parameters
• Translation export for manual inspection and error analysis
• Performance comparison between different model checkpoints

Typical Workflow:

1. Load your fine-tuned model from training output directory
2. Configure evaluation datasets and metrics
3. Run batch inference on test sets
4. Generate comprehensive evaluation reports
5. Export translations for qualitative analysis

Model Training

Neural Machine Translation Training

File: re-train.py

Comprehensive training pipeline for sequence-to-sequence translation models.

# Single experiment
python re-train.py --experiment_dir ./experiments-1 --experiments 36

# Multiple experiments
python re-train.py --experiment_dir ./experiments-1 --experiments 36 37 38

# All experiments
python re-train.py --experiment_dir ./experiments-1

Configuration via JSON files:

Create experiment configuration files in your experiment directory:

{
    "MODEL_NAME_OR_PATH": "Helsinki-NLP/opus-mt-en-ru",
    "TRAIN_DATASET_NAME": "sethjsa/medline_en_ru_parallel",
    "VALID_DATASET_NAME": null,
    "TEST_DATASET_NAME": null,
    "SOURCE_LANG": "en",
    "TARGET_LANG": "ru",
    "MAX_SOURCE_LENGTH": 128,
    "MAX_TARGET_LENGTH": 128,
    "LEARNING_RATE": 2e-5,
    "PER_DEVICE_TRAIN_BATCH_SIZE": 16,
    "PER_DEVICE_EVAL_BATCH_SIZE": 16,
    "NUM_TRAIN_EPOCHS": 3,
    "OUTPUT_DIR": "./results-custom"
}

Features:

• Automatic train/validation/test splitting when only training data provided
• Support for merging multiple training datasets
• BLEU score evaluation during training
• TensorBoard logging
• Sample text generation callbacks
• Reproducible training with fixed seeds

Dataset Handling:

• Single dataset: Automatically splits 80% train, 10% validation, 10% test
• Multiple datasets: Merges all specified training datasets
• Separate datasets: Uses provided train/validation/test splits

Alternative: Interactive Training

You can also fine-tune models interactively using the Jupyter notebook:

jupyter notebook train.ipynb

The notebook provides a step-by-step interface for model training with real-time monitoring and customization options.

Domain Classification

Medical vs Non-Medical Text Classification

File: domain_classifier_train.py

Trains a multilingual classifier to distinguish medical from non-medical text in both Russian and English.

python domain_classifier_train.py \
    --model_checkpoint xlm-roberta-base \
    --batch_size 8 \
    --learning_rate 2e-5 \
    --num_epochs 3 \
    --max_length 128

Training Data:

• Medical (Russian): sethjsa/medline_ru_mono
• Medical (English): sethjsa/emea_en_mono
• Non-Medical (Russian): sagteam/cedr_v1
• Non-Medical (English): fancyzhx/ag_news

Command Line Arguments:

• --model_checkpoint: Base model (default: xlm-roberta-base)
• --batch_size: Training batch size (default: 8)
• --learning_rate: Learning rate (default: 2e-5)
• --num_epochs: Training epochs (default: 3)
• --max_length: Maximum sequence length (default: 128)
• --train_size: Training set proportion (default: 0.8)
• --val_size: Validation set proportion (default: 0.1)

Output:

• Trained model pushed to Hugging Face Hub
• Classification performance on test set
• Sample inference examples

Diagnostic Analysis

LLM Backtranslation Diagnostics

File: llm_bt_diagnostics.py

Comprehensive diagnostic analysis of synthetic translation data quality.

python llm_bt_diagnostics.py \
    --human-ds sethjsa/medline_en_ru_parallel \
    --synthetic-ds giannhskp/medline_ru_en_llm-backtranslation \
                   giannhskp/scipar_ru_en_llm-backtranslation \
    --eval-ds sethjsa/wmt_biomedical_en_ru_test \
    --lm mistralai/Mistral-7B-v0.3 \
    --device cuda:0

Three Diagnostic Metrics:

1. Perplexity Analysis (1A):

   • Measures how "natural" synthetic text appears to a language model
   • Lower perplexity indicates more natural-sounding text
   • Numerically stable computation with loss clamping

2. Lexical Novelty (1B):

   • Computes 4-gram novelty ratios compared to human text
   • Measures how much synthetic text differs from human references
   • Higher novelty may indicate less human-like patterns

3. Embedding Distance (1C):

   • Cosine distance between synthetic and evaluation text embeddings
   • Uses sentence transformers for semantic similarity
   • Measures semantic closeness to target domain

Command Line Options:

• --human-ds: Reference human dataset
• --synthetic-ds: List of synthetic datasets to analyze
• --eval-ds: Evaluation dataset for embedding comparison
• --lm: Language model for perplexity computation
• --embedder: Sentence transformer model
• --device: Computation device (cpu/cuda)
• --max-ce: Maximum cross-entropy before clamping
• --filter-gt5: Only analyze sentences >5 words

Outputs:

• output_llm-bt_diagnostics/ppl_<dataset>.csv: Perplexity scores
• output_llm-bt_diagnostics/novelty_<dataset>.csv: Novelty ratios
• output_llm-bt_diagnostics/dist_<dataset>.csv: Embedding distances
• Console summary tables with statistics

File Reference

Core Scripts

Notebook Files

Workflow Examples

Complete Backtranslation Pipeline

1. Generate LLM backtranslations:

python llm-backtranslation-Tower.py

2. Evaluate quality:

python evaluate-bt-quality.py

3. Run diagnostics:

python llm_bt_diagnostics.py \
    --human-ds sethjsa/medline_en_ru_parallel \
    --synthetic-ds giannhskp/medline_ru_en_llm-backtranslation \
    --eval-ds sethjsa/wmt_biomedical_en_ru_test

4. Train translation model:

python re-train.py --experiment_dir ./experiments --experiments 1

Comprehensive Data Filtering Workflow

1. Train domain classifier:

python domain_classifier_train.py --num_epochs 3

2. Apply quality filtering:

python custom_utils/filter.py \
    --input_dataset "giannhskp/medline_ru_en_llm-backtranslation" \
    --threshold 0.4 \
    --hf_username "your_username" \
    --use_gpu

3. Apply domain classification filtering:

python custom_utils/filter_with_classifier.py \
    --dataset_names "your_username/medline_ru_en_llm-backtranslation_filtered_40_wmt22-cometkiwi-da" \
    --model_name_or_path "your_username/xlm-roberta-base_domain_classifier" \
    --text_column "en" \
    --label_to_keep 1 \
    --push_to_hub \
    --hub_repo_prefix "your_username/classifier-filtered-"

4. Apply length filtering:

jupyter notebook dataset_notebooks/filter_by_length.ipynb
# Select filtered dataset and apply length filter

5. Evaluate filtered data quality:

python evaluate-bt-quality.py

Data Augmentation for Robustness Testing

1. Create corrupted datasets:

jupyter notebook dataset_notebooks/corrupt_data.ipynb
# Configure corruption parameters and select target datasets

2. Train model with corrupted data:

python re-train.py --experiment_dir ./experiments-robust --experiments 1

3. Evaluate robustness:

python evaluate-llm.py  # Test on standard benchmarks

4. Run diagnostic analysis:

python llm_bt_diagnostics.py \
    --human-ds sethjsa/medline_en_ru_parallel \
    --synthetic-ds "your_username/dataset_wd_1_0.15" \
    --eval-ds sethjsa/wmt_biomedical_en_ru_test

End-to-End Research Pipeline

1. Data Generation:

# Traditional backtranslation
jupyter notebook original_backtranslation.ipynb

# LLM backtranslation  
python llm-backtranslation-Tower.py

# Pivot backtranslation
python pivot.py

2. Data Quality Control:

# Multi-stage filtering
python custom_utils/filter.py --threshold 0.4
python custom_utils/filter_with_classifier.py --label_to_keep 1
jupyter notebook dataset_notebooks/filter_by_length.ipynb

3. Data Augmentation:

# Create robust training variants
jupyter notebook dataset_notebooks/corrupt_data.ipynb

4. Model Training:

# Train with different datasets
python re-train.py --experiment_dir ./experiments --experiments 1 2 3

5. Comprehensive Evaluation:

# Quality evaluation
python evaluate-bt-quality.py

# LLM evaluation
python evaluate-llm.py

# Interactive evaluation
jupyter notebook evaluation.ipynb

# Diagnostic analysis
python llm_bt_diagnostics.py