Ensemble PhoBERT & FastText in Vietnamese Sentiment Analysis task

UPDATE:

• Optimize code directory structure

TO DO:

• fix ensemble procedure
• Containerize with Docker

Dataset

I used UIT-VSFC (Vietnamese Students’ Feedback Corpus) dataset in this project. This dataset collected students feedback about the school after every semester from 2013 to 2016. It contains over 16,000 sentences with 2 tasks: sentiments and topics. In this project I just experimented with sentiment task only.

In sentimenst task, there are 3 labels: 0: Positive, 1:Neural and 2:Negative. The distriution of this task is highly imbalanced, most of the label in Positive and Negative which tremendously affect the performance.

Model

• Finetune PhoBERT on downstream task.
• Build a FastText embedding on train+val corpus, vector_dim=300.

I experimented on these models:

• PhoBERT(base/large) + FeedForward.
• PhoBERT(base/large) + LSTM.
• FastText + LSTM.
• FastText + SVM.

Experiment

• Use CrossEntropyLoss as loss function.
• Final Dense Layer used LogSoftmax as acivation function. Experiment shows that it not only helps training process more stable but also improves performance.
• Finetuning PhoBERT(large) on Google Colab always faced with OOM, I used Gradient Accumulation to fix this issue.
• Finetune PhoBERT with Adam optimizer, learning_rate=1e-4, apply OneCycleLR learning rate scheduler with max_lr=learning_rate.
• FastText_LSTM also use Adam optimizer, learning_rate=1e-3, apply OneCycleLR learning rate scheduler with max_lr=learning_rate.
• First experiment show on Confusion Matrix that majority of models's performance not good in predicting 1:Neural because of Imbalanced dataset.
• Ensemble model on first experiment does not significantly improve the model's performance.
• Then experiment with class_weight in order to due with class imbalanced improved Precision and F1-score on all models.
• Ensemble model with second experiment improve all models's performance.

Ensemble_pred= ratio * pred1 + (1-ratio) * pred2

• ratio in range[0, 1]

• pred1 is the model with superior performance, pred2 vice versa. Ensemble_pred compares result with pred1's model.

Directory Structure

Expect directory structure be like:

├── /config
├── /data
│   ├── README.txt
│   ├── dev
│   │   ├── sentiments.txt
│   │   ├── sents.txt
│   │   └── topics.txt
│   ├── test
│   │   ├── sentiments.txt
│   │   ├── sents.txt
│   │   └── topics.txt
│   └── train
│       ├── sentiments.txt
│       ├── sents.txt
│       └── topics.txt
├── /src
├── ensemble.py
├── main.py
├── requirements.txt
├── test.sh
├── train.sh
├── train_fasttext.sh
├── train_svm.sh
└── utils.py

data is the UIT-VSFC downloaded

Run code

Run the following command for execution information:

python main.py --help

1. Install dependencies

pip install -r requirements.txt

2. Train FastText Embedding

bash train_fasttext.sh

3. Train BERT-base models & FastText-LSTM

• Check hyperpamter at config folder, modify it as your need

bash train.sh

4. Test BERT-base models & FastText-LSTM

bash test.sh

Result

Evaluation on Test Set

Emsemble evaluation on Test Set

Evaluation on Test set with class weights

Ensemble Evaluation on Test set with class weights