Limited Effectiveness of LLM-based Data Augmentation for COVID-19 Misinformation Stance Detection

The repository contains scripts and data used in the paper Limited Effectiveness of LLM-based Data Augmentation for COVID-19 Misinformation Stance Detection.

As for Twitter/X data, we used a dataset collected, documented, and managed by Chen et al., 2020, Hossain et al., 2020, and Hou et al., 2022. To comply with Twitter/X’s Terms of Service, we are only publicly releasing the Tweet IDs of the collected Tweets. The data is released for non-commercial research use.

Before executing the scripts, retrieve the tweet text using the 'tweet_id' field from dataset.json and add a new 'tweet' field for each row in the same JSON file. Use the API documentation to fetch tweets by their IDs.

Script Descriptions:

• data_check.py: Identifies trivial and overlapping claim-tweet pairs as detailed in Section 2 DATASETS.
• pretrain.py: Pretrains a classifier using the SNLI, MNLI, and NLI-FEVER datasets. The pretrained model is saved under ./pretrained_ce.
• pretrain_test.py: Evaluates the pretrained classifier on the test set. Performance is reported under "No DA: Pretrained" in Table 2.
• augment_trad.py: Applies AEDA, EDA, and backtranslation to augment the 'tweet' field.
• augment_cmg.py [-m]: Conducts CMG augmentation on the 'claim' field using three examples from the 'tweet' field. In practice, the OpenAI Batch API was used to reduce costs when running CMG on OpenAI models (GPT-4o, GPT-3.5-turbo).
• rejection_score.py: Computes the rejection score presented in Figure 3.
• train.py: Finetunes models with or without augmented data and saves all trained models under ./models.
• test.py: Evaluates the finetuned models, with results shown in Table 2.

This dataset is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Public License (CC BY-NC-SA 4.0). By using this dataset, you agree to abide by the stipulations in the license, remain in compliance with Twitter’s Terms of Service, Meta's Llama 2 Community License Agreement, Tongyi Qianwen's License Agreement, and OpenAI's Terms of Use, and cite the following manuscript:

Eun Cheol Choi, Ashwin Balasubramanian, Jinhu Qi, Emilio Ferrara. 2025. Limited effectiveness of LLM-based data augmentation for COVID-19 misinformation stance detection. In Companion Proceedings of the ACM Web Conference 2025 (WWW ’25 Companion), Apr 28–May 2, 2025, Sydney, Australia. ACM, New York, NY, USA, 4 pages. https://doi.org/10.1145/3701716.3715521

BibTeX:

@inproceedings{choi2025limited,
  title={Limited effectiveness of LLM-based data augmentation for COVID-19 misinformation stance detection},
  author={Choi, Eun Cheol and Balasubramanian, Ashwin and Qi, Jinhu and Ferrara, Emilio},
  booktitle={Companion Proceedings of the ACM on Web Conference 2025},
  year={2025}
}