README
BERT5urk
This repository hosts the new 1.42B Turkish T5 model named BERT5urk.
BERT5urk is part of the Turkish Model Zoo family and pretrained using the awesome T5X library with the UL2 objective.
Inspired by the great Finnish T5 and UL2 models from the Finnish NLP group, BERT5urk also uses UL2 and the efficient T5 architecture, that is proposed in the "Scale Efficiently" paper. Many thanks to the Finnish NLP group for open-sourcing the pretraining code and models!
Pretraining Data
BERT5urk uses the Turkish part of the amazing FineWeb2 corpus. Only documents with a higher language score than 0.99 are chosen for final pretraining corpus, that has a total size of 262GB.
We train a SPM-based vocab on a 3GB corpus from randomly chosen documents of the pretraining corpus.
Pretraining
BERT5urk was pretrained with the awesome T5X library. Some pretraining highlights:
- One-shot pretraining (pretraining without any training crashes) was possible a v3-32 TPU Pod and took 16.56 days
- Model was pretrained for 2M steps for an input & output sequence length of 512 and a batch size of 128
- The resulting model has 1.42B parameters
Evaluation
Detailed evaluations can be found in the Turkish Model Zoo repository. Additionally, we also fine-tuned TURNA models as it is another T5 model with 1.14B parameters for comparison.
Encoder-only Results
For experiments on named entity recognition (NER) and part-of-speech (PoS) tagging we also the awesome Flair library and fine-tune only the encoder of BERT5urk and TURNA. The overall performance can be seen in the following table:
| Model Name | Overall Development | Overall Test |
|---|---|---|
| BERTurk (cased, 128k) | 89.72 | 90.05 |
| BERTurk (uncased, 128k) | 89.25 | 89.95 |
| BERTurk (cased, 32k) | 88.98 | 89.49 |
| BERTurk (uncased, 32k) | 89.28 | 89.67 |
| ConvBERTurk (cased) | 90.06 | 90.27 |
| ConvBERTurk mC4 (cased) | 90.03 | 90.09 |
| ConvBERTurk mC4 (uncased) | 89.76 | 89.97 |
| DistilBERTurk (cased) | 87.95 | 88.16 |
| ELECTRA Base (cased) | 89.08 | 89.91 |
| ELECTRA Base mC4 (cased) | 89.24 | 90.03 |
| ELECTRA Base mC4 (uncased) | 89.09 | 89.62 |
| ELECTRA Small (cased) | 87.27 | 88.28 |
| BERT5urk | 89.96 | 90.26 |
| TURNA | 88.81 | 89.36 |
Encoder-decoder Results
We tried to replicate the results from the TURNA paper using the TURNA fine-tuning library.
Paraphrasing - Tatoeba
We fine-tune five different models for both TURNA and BERT5urk with different seeds and report the average score. Additionally the score from the TURNA paper is also shown in the following table:
| Model | test_rouge1 | test_rouge2 | test_rougeL | test_bleu | test_meteor |
|---|---|---|---|---|---|
| TURNA (paper) | 90.22 | 80.23 | 88.95 | 71.14 | 87.56 |
| TURNA (replicated) | 90.36 | 80.50 | 89.10 | 71.48 | 87.63 |
| BERT5urk | 90.47 | 80.78 | 89.21 | 71.89 | 87.74 |
Paraphrasing - OpenSubtitles
We fine-tune TURNA and BERT5urk only for one seed (due to resource limitations) and report scores (incl. scores from the TURNA paper):
| Model | test_rouge1 | test_rouge2 | test_rougeL | test_bleu | test_meteor |
|---|---|---|---|---|---|
| TURNA (paper) | 78.43 | 63.58 | 76.81 | 51.47 | 74.79 |
| TURNA (replicated) | 78.36 | 63.42 | 76.71 | 51.39 | 74.94 |
| BERT5urk | 78.56 | 63.80 | 76.95 | 51.74 | 75.07 |
Title Generation - TrNews
We fine-tune TURNA and BERT5urk only for one seed (due to resource limitations) and report scores (incl. scores from the TURNA paper):
| Model | test_rouge1 | test_rouge2 | test_rougeL | test_bleu | test_meteor |
|---|---|---|---|---|---|
| TURNA (paper) | 36.47 | 22.88 | 35.47 | 12.64 | 23.62 |
| TURNA (replicated) | 41.65 | 27.60 | 36.77 | 18.60 | 34.55 |
| BERT5urk | 41.79 | 27.77 | 37.00 | 19.08 | 34.69 |
Summarization - TrNews
We fine-tune TURNA and BERT5urk only for one seed (due to resource limitations) and report scores (incl. scores from the TURNA paper):
| Model | test_rouge1 | test_rouge2 | test_rougeL | test_bleu | test_meteor |
|---|---|---|---|---|---|
| TURNA (paper) | 41.77 | 27.81 | 36.99 | 19.05 | 34.61 |
| TURNA (replicated) | 40.75 | 26.82 | 35.88 | 18.00 | 33.91 |
| BERT5urk | 41.00 | 27.08 | 36.24 | 18.78 | 23.96 |
Acknowledgments
Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC). Many Thanks for providing access to the TPUs over many years ❤️
Made from Bavarian Oberland with ❤️ and 🥨.
Contributors
14 commits
Similar repos
README
BERT5urk
This repository hosts the new 1.42B Turkish T5 model named BERT5urk.
BERT5urk is part of the Turkish Model Zoo family and pretrained using the awesome T5X library with the UL2 objective.
Inspired by the great Finnish T5 and UL2 models from the Finnish NLP group, BERT5urk also uses UL2 and the efficient T5 architecture, that is proposed in the "Scale Efficiently" paper. Many thanks to the Finnish NLP group for open-sourcing the pretraining code and models!
Pretraining Data
BERT5urk uses the Turkish part of the amazing FineWeb2 corpus. Only documents with a higher language score than 0.99 are chosen for final pretraining corpus, that has a total size of 262GB.
We train a SPM-based vocab on a 3GB corpus from randomly chosen documents of the pretraining corpus.
Pretraining
BERT5urk was pretrained with the awesome T5X library. Some pretraining highlights:
- One-shot pretraining (pretraining without any training crashes) was possible a v3-32 TPU Pod and took 16.56 days
- Model was pretrained for 2M steps for an input & output sequence length of 512 and a batch size of 128
- The resulting model has 1.42B parameters
Evaluation
Detailed evaluations can be found in the Turkish Model Zoo repository. Additionally, we also fine-tuned TURNA models as it is another T5 model with 1.14B parameters for comparison.
Encoder-only Results
For experiments on named entity recognition (NER) and part-of-speech (PoS) tagging we also the awesome Flair library and fine-tune only the encoder of BERT5urk and TURNA. The overall performance can be seen in the following table:
| Model Name | Overall Development | Overall Test |
|---|---|---|
| BERTurk (cased, 128k) | 89.72 | 90.05 |
| BERTurk (uncased, 128k) | 89.25 | 89.95 |
| BERTurk (cased, 32k) | 88.98 | 89.49 |
| BERTurk (uncased, 32k) | 89.28 | 89.67 |
| ConvBERTurk (cased) | 90.06 | 90.27 |
| ConvBERTurk mC4 (cased) | 90.03 | 90.09 |
| ConvBERTurk mC4 (uncased) | 89.76 | 89.97 |
| DistilBERTurk (cased) | 87.95 | 88.16 |
| ELECTRA Base (cased) | 89.08 | 89.91 |
| ELECTRA Base mC4 (cased) | 89.24 | 90.03 |
| ELECTRA Base mC4 (uncased) | 89.09 | 89.62 |
| ELECTRA Small (cased) | 87.27 | 88.28 |
| BERT5urk | 89.96 | 90.26 |
| TURNA | 88.81 | 89.36 |
Encoder-decoder Results
We tried to replicate the results from the TURNA paper using the TURNA fine-tuning library.
Paraphrasing - Tatoeba
We fine-tune five different models for both TURNA and BERT5urk with different seeds and report the average score. Additionally the score from the TURNA paper is also shown in the following table:
| Model | test_rouge1 | test_rouge2 | test_rougeL | test_bleu | test_meteor |
|---|---|---|---|---|---|
| TURNA (paper) | 90.22 | 80.23 | 88.95 | 71.14 | 87.56 |
| TURNA (replicated) | 90.36 | 80.50 | 89.10 | 71.48 | 87.63 |
| BERT5urk | 90.47 | 80.78 | 89.21 | 71.89 | 87.74 |
Paraphrasing - OpenSubtitles
We fine-tune TURNA and BERT5urk only for one seed (due to resource limitations) and report scores (incl. scores from the TURNA paper):
| Model | test_rouge1 | test_rouge2 | test_rougeL | test_bleu | test_meteor |
|---|---|---|---|---|---|
| TURNA (paper) | 78.43 | 63.58 | 76.81 | 51.47 | 74.79 |
| TURNA (replicated) | 78.36 | 63.42 | 76.71 | 51.39 | 74.94 |
| BERT5urk | 78.56 | 63.80 | 76.95 | 51.74 | 75.07 |
Title Generation - TrNews
We fine-tune TURNA and BERT5urk only for one seed (due to resource limitations) and report scores (incl. scores from the TURNA paper):
| Model | test_rouge1 | test_rouge2 | test_rougeL | test_bleu | test_meteor |
|---|---|---|---|---|---|
| TURNA (paper) | 36.47 | 22.88 | 35.47 | 12.64 | 23.62 |
| TURNA (replicated) | 41.65 | 27.60 | 36.77 | 18.60 | 34.55 |
| BERT5urk | 41.79 | 27.77 | 37.00 | 19.08 | 34.69 |
Summarization - TrNews
We fine-tune TURNA and BERT5urk only for one seed (due to resource limitations) and report scores (incl. scores from the TURNA paper):
| Model | test_rouge1 | test_rouge2 | test_rougeL | test_bleu | test_meteor |
|---|---|---|---|---|---|
| TURNA (paper) | 41.77 | 27.81 | 36.99 | 19.05 | 34.61 |
| TURNA (replicated) | 40.75 | 26.82 | 35.88 | 18.00 | 33.91 |
| BERT5urk | 41.00 | 27.08 | 36.24 | 18.78 | 23.96 |
Acknowledgments
Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC). Many Thanks for providing access to the TPUs over many years ❤️
Made from Bavarian Oberland with ❤️ and 🥨.
Similar repos
Linked in READMEs
Contributors
14 commits