README
Visual Question Answering project
Dataset
Pipeline
CNN + LSTM approach
- Image Encoder: ResNet50
- Text Encoder: BiLSTM
Tranformers approach
- Image Encoder:
Vision Transformer,ViTMAE - Text Encoder:
RoBERTa base model
Train scripts
python train_vqa_basic_trainer.py \
--visual-pretrained "google/vit-base-patch16-224" \
--text-pretrained "roberta-base" \
--device "cuda:0"
Train results
| Models | Val acc | Test acc |
|---|---|---|
| ResNet50 + LSTM | 0.5358 | - |
| VisTrans + RoBERTa (pooler_output) | 0.6690 | 0.6636 |
| VisTrans + RoBERTa (last_hidden_state output) | 0.6931 | 0.6874 |
To do
- Public models
- Inference code
- Compare with other models
Contributors
hoannc0506
21 commits
Similar repos
Languages
Python
100.0%
README
Visual Question Answering project
Dataset
Pipeline
CNN + LSTM approach
- Image Encoder: ResNet50
- Text Encoder: BiLSTM
Tranformers approach
- Image Encoder:
Vision Transformer,ViTMAE - Text Encoder:
RoBERTa base model
Train scripts
python train_vqa_basic_trainer.py \
--visual-pretrained "google/vit-base-patch16-224" \
--text-pretrained "roberta-base" \
--device "cuda:0"
Train results
| Models | Val acc | Test acc |
|---|---|---|
| ResNet50 + LSTM | 0.5358 | - |
| VisTrans + RoBERTa (pooler_output) | 0.6690 | 0.6636 |
| VisTrans + RoBERTa (last_hidden_state output) | 0.6931 | 0.6874 |
To do
- Public models
- Inference code
- Compare with other models
Similar repos
Contributors
hoannc0506
21 commits
Languages
Python
100.0%