ParaDetox: Detoxification with Parallel Data (Russian). Paraphrase Task Negative Results

This repository contains information about Paraphrase Task markup from Russian Paradetox dataset collection pipeline.

ParaDetox Collection Pipeline

The ParaDetox Dataset collection was done via Yandex.Toloka crowdsource platform. The collection was done in three steps:

• Task 1: Generation of Paraphrases: The first crowdsourcing task asks users to eliminate toxicity in a given sentence while keeping the content.
• Task 2: Content Preservation Check: We show users the generated paraphrases along with their original variants and ask them to indicate if they have close meanings.
• Task 3: Toxicity Check: Finally, we check if the workers succeeded in removing toxicity.

Specifically this repo contains the results of Task 1: Generation of Paraphrases. The general size of the dataset is about 11,446 samples. Here, the samples that were marked by annotators that they cannot detoxify are present. The reason for this can be following:

• non-toxic: the text is simply non toxic, can be with negative sentiment, however, without any obscene or rude lexicon;
• toxic content: the text is passive aggressive, sarcastic, or other, so the insult is deeply incorporated in the message. To detoxify it, you need to change the meaning dramantically.
• unclear: the text is only about obscene lexicon, random words, or any other tokens combination that makes it difficult to understand the main content.

Annotators could select several options.

Citation

@inproceedings{logacheva-etal-2022-study,
    title = "A Study on Manual and Automatic Evaluation for Text Style Transfer: The Case of Detoxification",
    author = "Logacheva, Varvara  and
      Dementieva, Daryna  and
      Krotova, Irina  and
      Fenogenova, Alena  and
      Nikishina, Irina  and
      Shavrina, Tatiana  and
      Panchenko, Alexander",
    booktitle = "Proceedings of the 2nd Workshop on Human Evaluation of NLP Systems (HumEval)",
    month = may,
    year = "2022",
    address = "Dublin, Ireland",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.humeval-1.8",
    doi = "10.18653/v1/2022.humeval-1.8",
    pages = "90--101",
    abstract = "It is often difficult to reliably evaluate models which generate text. Among them, text style transfer is a particularly difficult to evaluate, because its success depends on a number of parameters.We conduct an evaluation of a large number of models on a detoxification task. We explore the relations between the manual and automatic metrics and find that there is only weak correlation between them, which is dependent on the type of model which generated text. Automatic metrics tend to be less reliable for better-performing models. However, our findings suggest that, ChrF and BertScore metrics can be used as a proxy for human evaluation of text detoxification to some extent.",
}

Contacts

For any questions, please contact: Daryna Dementieva (dardem96@gmail.com)