AI ZAMAN hack solution

by: Sk ML team

How to reproduce the solution

1. Set up the environment conda env create --file environment.yml conda activate tatar

2. Finetune the model: python ./script_models/finetune_mt0_tatar_lora.py

3. Run the detoxification python script_models/mt0_ranked_lexicon_penalty_v3.py --input ./input_data/test_inputs.tsv --out dev_mt0_ranked_lexicon_ft_ru_big.tsv --text-col tat_toxic --out-col tat_detox1 --lang tt --batch-size 8 --model ./mt0-xl-detox-orpo-tt-lora --n-candidates 32 --num-beams 32 --temperature 0.88 --top-p 0.81 --repetition-penalty 1.2 --sim-weight 1.14 --tox-weight 1.68 --lex-weight 0.63 --tox-threshold 0.3

4. Evaluate the result python ./scripts_metric/run_all_metrics.py --input ./dev_mt0_ranked_lexicon_ft_ru_big.tsv

Our computational resourses

We ran this pipeline on 1xNvidia-5090 & Intel Core Ultra 7 265 KF. It took ~10 minutes to recieve the results from scratch

Methodology

Our final solution was based on mT0-XL model. Also, we used XLM-R for toxicity score computation, LaBSE and X-COMET for semantic similarity.

The pipeline was the following:

1. First, we preprocessed texts and removed some bad words of Tatar language (we googled them)

2. Next we used MT0 to generate n=32 candidates (detoxified texts for each sample)

3. Each candidate was scored by several criteria. The final score was the weighted sum of the following scores.

   • toxicity score by XLM-R
   • semantic similarity between the original text and candidates by LaBSE and X-COMET
   • lexicon penalty (how many bad words from the lexicon (bad_words) appear in the text)
   • length penalty (if the candidate is shorter than the minimum allowed ratio)

4. The best candidate was taken as a final answer for the sample

This pipeline was quite stable both on dev and test. But the final version included fine-tuning of the MT0 on its own answers for dev data. This fine-tuned model was applied to the test data then and we got the score of 0.67 on the test set.

We tuned hyperparameters such as weights for scoring, temperature, n_candidates, num_beams, top_p using latin hypercube algorithm (./script_models/tune_mt0_tatar.py)