Multimodal deep learning model for fake news recognition

This is the code for the multimodal model that tackles the problem of fake news detection. It obtained the first place in the MULTI-Fake-DetectiVE competition proposed in the workshop EVALITA 2023.

The research paper regarding this work will be available soon!

The Problem

The task proposed asks to classify a piece of information composed of a text and one or more images in four classes:

• Certainly Fake
• Probably Fake
• Probably Real
• Certainly Real

The proposed solution

First of all, the texts in the dataset were of arbitrary length. Since the NLP models like BERT present a limit to how long a text can be, a way to process longer texts was found in literature and implemented. The text was transformed in tokens, then divided in pieces that could be processed by BERT and the [CLS] tokens of the result of the single pieces were averaged to find a final representation for the text.

Several tests were conducted in order to find the best model. In the end what worked out best was to re-implement as base an existing model called FND-CLIP, enhanced with some original extensions. The base model includes 3 flows of information:

1. The visual information one, achieved by the concatenation of the embeddings obtained by ResNet and the visual encoder of CLIP
2. The textual information one, achieved by the cooncation of obtained by BERT and the textual encoder of CLIP
3. The mixed one, achieved by the cooncation of obtained by the visual encoder of CLIP and the textual encoder of CLIP

The embeddings are then processed by some FC layers, weighted by Squeeze-and-excitation layers and then summed to be finally classified by a classifier.

The extensions applied are:

• A. Sentiment Analysis: Usage of the embeddings extracted with BERT pretrained for sentiment analysis. It includes two sub-cases:
        A1. The sentiment embeddings are concatenated with the textual flow
        A2. The sentiment embeddings constitute a separate flow
• B. Images frequency domain analysis: Images generated from other AI often produce artifacts that can be easily spotted in the frequency domain. So, this extension uses the Discrete Fourier Transform on the images to go from the spatial domain to the frequency domain. The real and imaginary parts obtained are both processed by a VGG19 net and then concatenated to form another flow.
• C. Concatenation of the embeddings: Instead of summing the flow embeddings at the end, I decided to concatenate them in order to give the network more possibility to gather the relationship between the different modalities.
• D. Back translation: Since the dataset presents classes that are not balanced, the underrepresented classes were augmented with the process of back-translation, that consists in the translation in a language and back in the original language, in order to maintain the semantics while changing the words.
• E. Squeeze and excitation layers: Insted of using the Squeeze and excitation layers only at the end of the network, I tested their usage at the start, on the embeddings that will form the flow, but before the concatenation. This will help the network to learn how to weight the embeddings of the same modality to reduce overfitting.
• F. Focus on Region of Interest: Using [Detectron2](https://github.com/facebookresearch/detectron2) I extracted the embeddings of the region of interest from the images, averaged them and concatened the result with the visual flow.

Results

The metrics used are Accuracy, Precision, Recall, F1-score, F1-weighted. The main metric used is the F1-weighted since the dataset was not balanced. The extensions were used by themselves and combined. The best performance between all the tested combinations was obtained with the combination A1, C, D.
The overall best performance was however obtained by a weighed ensemble between the combinations:

• A2, B
• A1, E, D
• A1, C, D

The table of these results is presented below:

The weighted ensemble model won the first place in the competition. The table is taken from the official website and here reported:

How to run

Unfortunately, due to copyright, I cannot share the dataset. If you have your own dataset you can train the models inside the directory models. For example the model a1_c_d.py. To train it: python a1_c_d.py loss learning_rate save_path weight_decay tsvpath mediapath pretrained_bert_path where:

• loss: can be either "cross_entropy" or "focal"
• learning_rate: value of the learning rate to use
• save_path: file to save the best perfroamnce model
• weight_decay: value of the weight decay
• tsvpath: path to the tsv file of the dataset
• mediapath: path of the image folder of the dataset
• pretrained_bert_path: path to the pretrained bert model, required to handle long texts

To test the model, go in the evaluation directory and run the evaluation: python eval_model.py tsvpath mediapath model_path model_type where:

• tsvpath: path to the tsv file of the dataset
• mediapath: path of the image folder of the dataset
• model_path: path to the model to evaluate
• model_type: type of the model, can be either "a2_b", "a1_e_d", "a1_c_d"

To test the ensemble model: python eval_ensemble.py tsvpath mediapath path1 path2 path3 where:

• tsvpath: path to the tsv file of the dataset
• mediapath: path of the image folder of the dataset
• path1: path to the model "a2_b"
• path2: path to the model "a1_e_d"
• path3: path to the model "a1_c_d"