Kaggle CommonLit Readability Prize 2nd place solution

This is repository of the 2nd place solution of Kaggle CommonLit Readability Prize.
The discription of this solution is available here.
The prediction notebook in the competition is available here.
A partially revised prediction notebook is here.

Hardware

I used three different machines.

1. Google Cloud Platform

• Debian 10.10
• n1-highmem-8 (vCPU x 8, memory 52 GB)
• 1 x NVIDIA Tesla V100

2. Google Cloud Platform

• Ubuntu 18.04
• a2-highgpu-1g (vCPU x 12, memory 85 GB)
• 1 x NVIDIA Tesla A100

3. kaggle notebook

Environment(except for kaggle notebook)

$ docker-compose up --build -d
$ docker exec -it commonlit bash

Data download

Plese download data to ./data from https://www.kaggle.com/c/commonlitreadabilityprize/data and unzip it.

Model download

I used several pretrained models that are publicly available in the kaggle dataset. Please download it to the following directories and unzip it.

• ./models/roberta : https://www.kaggle.com/maroberti/roberta-transformers-pytorch
• ./models/bart : https://www.kaggle.com/xhlulu/bart-models-hugging-face-model-repository
• ./models/electra : https://www.kaggle.com/xhlulu/electra
• ./models/deberta : https://www.kaggle.com/xhlulu/deberta
• ./models/xlnet : https://www.kaggle.com/irustandi/xlnet-pretrained-models-pytorch

Prerocess

Plese download data to ./data from https://www.kaggle.com/takoihiraokazu/commonlit-fold and unzip it. This is the data for cross validation, and is the result of re-running here.

Train

$ sh bin/train.sh

Please note that the results of ex131.py(deberta-v2-xlarge), ex194.py(deberta-v2-xlarge), ex216.py(deberta-v2-xxlarge) and ex407.py(funnel-transformer-large) will change with each train. Also, I did not fix the seed in mlm_roberta_base.py during the competition period, so if you want to reproduce ex237.py, please use this pretrained model instead of output of mlm_roberta_base.py. I have also listed below the hardware numbers where I performed each training during the competition.

Ensemble & Post process

$ sh bin/ensemble_postprocess.sh

The ensemble weights were adjusted based on the output of ensemble.py and by looking at the Public Score. The PostProcess coefficients and thresholds were also adjusted based on the output of postprocess.py and by looking at the Public Score. The final weights, coefficientsare and thresholds as follows. The public and private scores for each model are also listed.

ensebmle

post process

• Predictions >= 0.3 -> Predictions * 1.07
• 0.3 > Predictions >= 0 -> Predictions * 1.2
• 0 > Predictions >= -0.7 -> Predictions * 0.97485814
• -0.7 > Predictions >= -0.9 -> Predictions * 1.01
• -0.9 > Predictions >= -2.0 -> Predictions * 1.02150304
• -2.0 > Predictions -> Predictions * 1.02764047

Predict

$ sh bin/predict.sh