Text Detoxification 2025

Overview

This repository contains solution for detoxification2025.

Usage

Because of shorted time, the solution is implemented notebook style, which is not suitable for production. However, it can be used for educational purposes and to understand the detoxification process.

The requrements are install in each notebook, so you can run them independently. The notebooks are designed to be run in order, but you can also run them individually.

Notebooks

• data_augment.ipynb: This notebook is used to augment the for 6 unseen languages, Using machine translation.
• data-processing.ipynb: This notebook is used to gather all data source, do the processing that require for fineturn and evaluation.
• finetune.ipynb: This notebook is used to finetune the model using the processed data.
• evaluate.ipynb: This notebook is used to evaluate/inference the model using the finetuned model.
• compute_metrics.ipynb: This notebook is used to do the post-process and compute the metrics for the evaluation.