Project Overview

Project Title: Text Detox

This project aims to develop a text detoxification tool that filters out harmful or toxic content from text inputs. The goal is to create a user-friendly application that can be integrated into various platforms to improve online communication.

Repository Structure

The structure of this repository is as follows:

COLX_531_TextDetox/
├── README.md
├── TextDetox_final_report.pdf
├── evaluation_script/
│   └── evaluate.py
├── milestone1/
│   ├── data_inspection.md
│   ├── img/
│   │   └── dataset_statistics.png
│   ├── milestone1.md
│   ├── progress_report.pdf
│   ├── proof_of_project_registration.pdf
│   └── teamwork_contract.md
├── milestone2/
│   ├── baseline/
│   │   ├── backtranslation_baseline.py
│   │   ├── Dockerfile
│   │   ├── en_baseline_input.jsonl
│   │   ├── predictions.jsonl
│   │   ├── README.md
│   │   ├── requirements.txt
│   │   └── run.sh
│   ├── llm_model/
│   │   ├── gold_extraction.py
│   │   ├── llm_model.py
│   │   └── wandb_log.py
│   └── progress_report_milestone2.pdf
├── milestone3/
│   └── progress_report_milestone3.pdf
├── milestone4/
│   ├── detoxification_input.jsonl
│   ├── finetuned_baseline.py
│   ├── golden_references.jsonl
│   ├── metrics.jsonl
│   ├── output.jsonl
│   ├── progress_report_milestone4.pdf
│   ├── README.md
│   └── wandb_log.py
├── milestone5/
│   ├── data/
│   │   ├── data_extraction.py
│   │   ├── golden_am.jsonl
│   │   ├── golden_en.jsonl
│   │   ├── golden_ru.jsonl
│   │   ├── input_am.jsonl
│   │   ├── input_en.jsonl
│   │   ├── input_ru.jsonl
│   │   ├── output_am.jsonl
│   │   ├── output_en.jsonl
│   │   └── output_ru.jsonl
│   ├── evaluation/
│   │   ├── evaluate.py
│   │   ├── evaluate_am.txt
│   │   ├── evaluate_en.txt
│   │   └── evaluate_ru.txt
│   ├── new_model.py
│   └── progress_report_milestone5.pdf
├── milestone6/
│   ├── data/
│   │   ├── am/
│   │   │   ├── output_am.txt
│   │   │   ├── test_am.jsonl
│   │   │   └── test_output_am.jsonl
│   │   ├── en/
│   │   │   ├── output_en.txt
│   │   │   ├── test_en.jsonl
│   │   │   └── test_output_en.jsonl
│   │   ├── fr/
│   │   │   ├── output_fr.txt
│   │   │   ├── test_fr.jsonl
│   │   │   └── test_output_fr.jsonl
│   │   ├── hin/
│   │   │   ├── output_hin.txt
│   │   │   └── test_hin.jsonl
│   │   ├── ru/
│   │   │   ├── output_ru.txt
│   │   │   ├── test_ru.jsonl
│   │   │   └── test_output_ru.jsonl
│   │   ├── zh/
│   │   │   ├── output_zh.txt
│   │   │   ├── test_zh.jsonl
│   │   │   └── test_output_zh.jsonl
│   │   └── data_extraction.py
│   ├── src/
│   │   ├── evaluate.py
│   │   ├── model_main.py
│   │   ├── train_classifier.py
│   │   ├── train_detox.py
│   │   └── wandb_log.py
│   └── progress_report_milestone6.pdf
├── src/
│   ├── data_inspection.ipynb
│   └── data_loader.py

Milestone 1 Overview

Milestone 1 focuses on the initial setup of the project, including the following key components:

• Progress Report: Documents the work completed during this milestone.
• Proof of Registration: Provides documentation of project registration.
• Teamwork Contract: Outlines team roles and responsibilities.
• Milestone Overview: Summarizes the objectives and outcomes for this milestone.

Milestone 1 Contents

• Progress Report: progress_report.pdf - A summary of the work completed during this milestone.
• Proof of Registration: proof_of_registration.pdf - Documentation of our project registration.
• Teamwork Contract: teamwork_contract.md - Agreement outlining team roles and responsibilities.
• Milestone Overview: milestone1.md - Detailed overview of the milestone objectives and outcomes.
• Data Inspection: data_inspection.md - Samples of inputs and outputs related to our data.
• Dataset Statistical Analysis: dataset_statistics.png - A table for dataset statistical analysis.
• Data Inspection Notebook: data_inspection.ipynb - Jupyter Notebook for loading data samples.

Milestone 2 Overview

Milestone 2 focuses on the development of the baseline and LLM models for text detoxification. Key components include:

• Baseline Model: Implementation of a backtranslation baseline approach.
• LLM Model: Development of a large language model for better detoxification results.

Milestone 2 Contents

• Baseline Directory: Contains files related to the baseline model.

  • Backtranslation Baseline: backtranslation_baseline.py - Script for the backtranslation baseline model.
  • Dockerfile: Dockerfile - Dockerfile for containerizing the baseline model.
  • Input Data: en_baseline_input.jsonl - Input data for the baseline model.
  • Predictions: predictions.jsonl - Output predictions from the baseline model.
  • Baseline README: README.md - Documentation for the baseline model.
  • Requirements: requirements.txt - List of dependencies for the baseline model.
  • Run Script: run.sh - Script to run the baseline model.

• LLM Model Directory: Contains files related to the LLM model.

  • LLM Model Implementation: llm_model.py - Script for the LLM model implementation.
  • Gold Extraction: gold_extraction.py - Script for extracting gold standard data.
  • WandB Logging: wandb_log.py - Script for logging experiments with Weights & Biases.

Milestone 3 Overview

Milestone 3 focuses on reflections on the previous milestones.

Milestone 3 Contents

• Progress Report: progress_report_milestone3.pdf - A summary of developments, challenges, and reflections made during this milestone.

---

Milestone 4 Overview

Milestone 4 focuses on our second model, including model improvements, evaluation outputs, results comparison and error analysis.

Milestone 4 Contents

• Milestone Overview: README.md - Summary of the final milestone objectives and outcomes.
• Progress Report: progress_report_milestone4.pdf - A detailed report on the development system, model description, results and error analysis.
• Fine-Tuned Model: finetuned_baseline.py - Implementation of the fine-tuned baseline model.
• Evaluation Metrics: metrics.jsonl - JSON file containing evaluation results and performance metrics.
• Detoxification Input: detoxification_input.jsonl - Input data used for the detoxification model.
• Golden References: golden_references.jsonl - Reference outputs for evaluating model performance.
• Model Outputs: output.jsonl - Outputs generated by the fine-tuned model.
• Logging and Tracking: wandb_log.py - Logging script for tracking experiments and results using Weights & Biases.

---

Milestone 5 Overview

Milestone 5 centers on the development and evaluation of our third model, incorporating multilingual capabilities and refined detoxification logic. This milestone includes the implementation of a new model, testing across English, Russian, and Amharic datasets, and a comprehensive analysis of results.

Milestone 5 Contents

• Progress Report: progress_report_milestone5.pdf - A detailed report covering the development process, model description, results, error analysis, and reflections on planning improvements.
• New Model Implementation: new_model.py - The script implementing the new detoxification model designed for this milestone.
• Data Directory:
  • Data Extraction: data_extraction.py - Script for extracting and preparing multilingual datasets.
  • Golden References:
    • golden_en.jsonl - Reference outputs for English dataset evaluation.
    • golden_ru.jsonl - Reference outputs for Russian dataset evaluation.
    • golden_am.jsonl - Reference outputs for Amharic dataset evaluation.
  • Input Data:
    • input_en.jsonl - Input data for English detoxification.
    • input_ru.jsonl - Input data for Russian detoxification.
    • input_am.jsonl - Input data for Amharic detoxification.
  • Model Outputs:
    • output_en.jsonl - Detoxified outputs for the English dataset.
    • output_ru.jsonl - Detoxified outputs for the Russian dataset.
    • output_am.jsonl - Detoxified outputs for the Amharic dataset.
• Evaluation Directory:
  • Evaluation Script: evaluate.py - Script for evaluating model performance across datasets.
  • Evaluation Results:
    • evaluate_en.txt - Evaluation metrics and analysis for the English dataset.
    • evaluate_ru.txt - Evaluation metrics and analysis for the Russian dataset.
    • evaluate_am.txt - Evaluation metrics and analysis for the Amharic dataset.

Milestone 6 Overview

Milestone 6 focuses on finalizing the detoxification pipeline with multilingual support and model evaluation across six languages: English, Amharic, Russian, Hindi, French, and Chinese. This milestone includes refined data processing, model evaluation, and comparison of outputs across languages.

Milestone 6 Contents

• Progress Report: progress_report_milestone6.pdf - A detailed report on the development system, model description, results and error analysis.

• Multilingual Model Outputs:

  • Amharic (am):
    • test_am.jsonl – Input data for Amharic.
    • test_output_am.jsonl – Model outputs.
    • output_am.txt – Formatted output for evaluation.
  • English (en):
    • test_en.jsonl
    • test_output_en.jsonl
    • output_en.txt
  • Russian (ru):
    • test_ru.jsonl
    • test_output_ru.jsonl
    • output_ru.txt
  • Hindi (hin):
    • test_hin.jsonl
    • output_hin.txt
  • French (fr):
    • test_fr.jsonl
    • test_output_fr.jsonl
    • output_fr.txt
  • Chinese (zh):
    • test_zh.jsonl
    • test_output_zh.jsonl
    • output_zh.txt
  • Data Extraction: data_extraction.py – Script for processing and preparing input/output JSONL files.

• Source Code Directory:

  • evaluate.py – Evaluation script to score model performance.
  • model_main.py – Main runner for the multilingual detox model.
  • train_classifier.py – Script for training toxicity classifiers.
  • train_detox.py – Script for training detoxification models.
  • wandb_log.py – Weights & Biases logging utility.

---

Milestone 7 Overview

Milestone 7 focuses on working with the final report and the presentation slides. This milestone works on generating final report for this entire project and includes the entire process of the model evolution and final result.

Milestone 7 Contents

• Final Report: TextDetox_final_report.pdf

---

Code Location

The required code for this project can be found in the following directories:

• src: Contains the source code files related to data inspection.
• evaluation_script: Directory for evaluation scripts (currently empty, official evaluation method to be announced).