This is a corpus of parallel Erzya-Russian words, phrases and sentences, collected in the paper The first neural machine translation system for the Erzya language.
Erzya (myv) is a language from the Uralic family. It is spoken primarily in the Republic of Mordovia and some other regions of Russia and other post-Soviet countries. We use the Cyrillic version of its script.
The corpus consists of the following parts:
| name | size | composition |
|---|---|---|
| train | 74503 | parallel words, phrases and sentences, mined from dictionaries, books and web texts |
| dev | 1500 | parallel sentences mined from books and web texts |
| test | 1500 | parallel sentences mined from books and web texts |
| mono | 333651 | Erzya sentences mined from books and web texts, translated to Russian by a neural model |
The dev and test splits contain sentences from the following sources
| name | size | description |
|---|---|---|
| wiki | 600 | Aligned sentences from linked Erzya and Russian Wikipedia articles |
| bible | 400 | Paired verses from the Bible (https://finugorbib.com) |
| games | 250 | Aligned sentences from the book "Сказовые формы мордовской литературы", И.И. Шеянова, 2017, НИИ гуманитарых наук при Правительстве Республики Мордовия, Саранск |
| tales | 100 | Aligned sentences from the book "Мордовские народные игры", В.С. Брыжинский, 2009, Мордовское книжное издательство, Саранск |
| fiction | 100 | Aligned sentences from modern Erzya prose and poetry (https://rus4all.ru/myv) |
| constitution | 50 | Aligned sentences from the Soviet 1938 constitution |
To load the first three parts (train, validation and test), use the code:
from datasets import load_dataset
data = load_dataset('slone/myv_ru_2022')
To load all four parts (included the back-translated data), please specify the data files explicitly:
from datasets import load_dataset
data_extended = load_dataset(
'slone/myv_ru_2022',
data_files={'train':'train.jsonl', 'validation': 'dev.jsonl', 'test': 'test.jsonl', 'mono': 'back_translated.jsonl'}
)
translation: the dataset may be used to train ru-myv translation models. There are no specific leaderboards for it yet, but if you feel like discussing it, welcome to the comments!The main part of the dataset (train, dev and test) consists of "natural" Erzya (Cyrillic) and Russian sentences, translated to the other language by humans. There is also a larger Erzya-only part of the corpus (mono), translated to Russian automatically.
All data instances have three string fields: myv, ru and src (the last one is currently meaningful only for dev and test splits), for example:
{'myv': 'Сюкпря Пазонтень, кие кирвазтизе Титэнь седейс тынк кисэ секе жо бажамонть, кона палы минек седейсэяк!',
'ru': 'Благодарение Богу, вложившему в сердце Титово такое усердие к вам.',
'src': 'bible'}
myv: the Erzya text (word, phrase, or sentence)ru: the corresponding Russian textsrc: the source of data (only for dev and test splits)This is, as far as we know, the first publicly available parallel Russian-Erzya corpus, and the first medium-sized translation corpus for Erzya. We hope that it sets a meaningful baseline for Erzya machine translation.
The dataset was collected from various sources (see below).
The texts were spit in sentences using the razdel package.
For some sources, sentences were filtered by language using the slone/fastText-LID-323 model.
For most of the sources, myv and ru sentences were aligned automatically using the slone/LaBSE-en-ru-myv-v1 sentence encoder
and the code from the paper repository.
The dataset comprises parallel myv-ru and monolingual myv texts from diverse sources:
No human annotation was involved in the data collection.
All data was collected from public sources, so no sensitive information is expected in them. However, some sentences collected, for example, from news articles or LiveJournal posts, can contain personal data.
Publication of this dataset may attract some attention to the endangered Erzya language.
Most of the dataset has been collected by automatical means, so it may contain errors and noise. Some types of these errors are systemic: for example, the words for "Erzya" and "Russian" are often aligned together, because they appear in the corresponding Wikipedias on similar positions.
The dataset is noisy: some texts in it may be ungrammatical, in a wrong language, or poorly aligned.
The data was collected by David Dale (https://huggingface.co/cointegrated).
The status of the dataset is not final, but after we check everything, we hope to be able to distribute it under the CC-BY-SA license.
[TBD]
This is a corpus of parallel Erzya-Russian words, phrases and sentences, collected in the paper The first neural machine translation system for the Erzya language.
Erzya (myv) is a language from the Uralic family. It is spoken primarily in the Republic of Mordovia and some other regions of Russia and other post-Soviet countries. We use the Cyrillic version of its script.
The corpus consists of the following parts:
| name | size | composition |
|---|---|---|
| train | 74503 | parallel words, phrases and sentences, mined from dictionaries, books and web texts |
| dev | 1500 | parallel sentences mined from books and web texts |
| test | 1500 | parallel sentences mined from books and web texts |
| mono | 333651 | Erzya sentences mined from books and web texts, translated to Russian by a neural model |
The dev and test splits contain sentences from the following sources
| name | size | description |
|---|---|---|
| wiki | 600 | Aligned sentences from linked Erzya and Russian Wikipedia articles |
| bible | 400 | Paired verses from the Bible (https://finugorbib.com) |
| games | 250 | Aligned sentences from the book "Сказовые формы мордовской литературы", И.И. Шеянова, 2017, НИИ гуманитарых наук при Правительстве Республики Мордовия, Саранск |
| tales | 100 | Aligned sentences from the book "Мордовские народные игры", В.С. Брыжинский, 2009, Мордовское книжное издательство, Саранск |
| fiction | 100 | Aligned sentences from modern Erzya prose and poetry (https://rus4all.ru/myv) |
| constitution | 50 | Aligned sentences from the Soviet 1938 constitution |
To load the first three parts (train, validation and test), use the code:
from datasets import load_dataset
data = load_dataset('slone/myv_ru_2022')
To load all four parts (included the back-translated data), please specify the data files explicitly:
from datasets import load_dataset
data_extended = load_dataset(
'slone/myv_ru_2022',
data_files={'train':'train.jsonl', 'validation': 'dev.jsonl', 'test': 'test.jsonl', 'mono': 'back_translated.jsonl'}
)
translation: the dataset may be used to train ru-myv translation models. There are no specific leaderboards for it yet, but if you feel like discussing it, welcome to the comments!The main part of the dataset (train, dev and test) consists of "natural" Erzya (Cyrillic) and Russian sentences, translated to the other language by humans. There is also a larger Erzya-only part of the corpus (mono), translated to Russian automatically.
All data instances have three string fields: myv, ru and src (the last one is currently meaningful only for dev and test splits), for example:
{'myv': 'Сюкпря Пазонтень, кие кирвазтизе Титэнь седейс тынк кисэ секе жо бажамонть, кона палы минек седейсэяк!',
'ru': 'Благодарение Богу, вложившему в сердце Титово такое усердие к вам.',
'src': 'bible'}
myv: the Erzya text (word, phrase, or sentence)ru: the corresponding Russian textsrc: the source of data (only for dev and test splits)This is, as far as we know, the first publicly available parallel Russian-Erzya corpus, and the first medium-sized translation corpus for Erzya. We hope that it sets a meaningful baseline for Erzya machine translation.
The dataset was collected from various sources (see below).
The texts were spit in sentences using the razdel package.
For some sources, sentences were filtered by language using the slone/fastText-LID-323 model.
For most of the sources, myv and ru sentences were aligned automatically using the slone/LaBSE-en-ru-myv-v1 sentence encoder
and the code from the paper repository.
The dataset comprises parallel myv-ru and monolingual myv texts from diverse sources:
No human annotation was involved in the data collection.
All data was collected from public sources, so no sensitive information is expected in them. However, some sentences collected, for example, from news articles or LiveJournal posts, can contain personal data.
Publication of this dataset may attract some attention to the endangered Erzya language.
Most of the dataset has been collected by automatical means, so it may contain errors and noise. Some types of these errors are systemic: for example, the words for "Erzya" and "Russian" are often aligned together, because they appear in the corresponding Wikipedias on similar positions.
The dataset is noisy: some texts in it may be ungrammatical, in a wrong language, or poorly aligned.
The data was collected by David Dale (https://huggingface.co/cointegrated).
The status of the dataset is not final, but after we check everything, we hope to be able to distribute it under the CC-BY-SA license.
[TBD]