MADLAD-400

Dataset and Introduction

MADLAD-400 (Multilingual Audited Dataset: Low-resource And Document-level) is a document-level multilingual dataset based on Common Crawl, covering 419 languages in total. This uses all snapshots of CommonCrawl available as of August 1, 2022. The primary advantage of this dataset over similar datasets is that it is more multilingual (419 languages), it is audited and more highly filtered, and it is document-level. The main disadvantage is also its strength -- being more filtered, it may lack the recall needed for some applications.

There are two versions released: the noisy dataset, which has no filtering except document-level LangID, and the clean dataset, which has a variety of filters applied, though it naturally has a fair amount of noise itself. Each dataset is released in a document-level form that has been deduplicated.

Loading

You can load both the clean and noisy versions of any language by specifing its LangID:

madlad_abt = load_dataset("allenai/madlad-400", "abt")

A list of langagues can also be supplied with a keyword argument:

madlad_multilang = load_dataset("allenai/madlad-400", languages=["abt", "ace"])

Additionally, you can load the noisy and clean subsets seperately with the split keyword argument:

madlad_multilang_clean = load_dataset("allenai/madlad-400", languages=["abt", "ace"], split="clean")

LangID model and Crawl

Following Language Id In the Wild, we trained a Semi-Supervised LangId model (SSLID) on 500 languages. The training data is as described in that paper, with the differences that 1) training data is sampled to a temperature of T=3 to reduce over-triggering on low-resource languages; and 2) the data is supplemented with web-crawled data from the same paper (that has already been through the various filters described therein) in the hopes that it will increase robustness to web-domain text.

Filtering

Before separating the raw CommonCrawl corpus by LangID, these filtering steps are done, similar to Raffel et al (2020):

• Discarded any page with fewer than 5 sentences and only retained lines that contained at least 3 words.
• Removed any line with the word Javascript.
• Removed any page where the phrase “lorem ipsum” appeared.
• Removed any pages containing the phrases "terms of use", "privacy policy", "cookie policy", "uses cookies", "use of cookies", "use cookies"
• Removed any pages that contained a curly bracket.
• To deduplicate the data set, discarded all but one of any three-sentence span occurring more than once in the data set.

The noisy subset of the data was filtered only by document-level LangID, which was taken to be the majority sentence-level LangID prediction. The clean subset removed all documents with a percent_questionable score greater than 20%. It furthermore removed any document with under 5 sentences.

The pct_questionable score is simple the percentage of sentences in the input document that were "questionable". A sentence was considered questionable if any of the following were true:

• LangID Consistency: the sentence-level LangID does not match the document-level LangID
• List Case: The sentence has at least 12 tokens, and over 50% percent of the tokens began in a capital letter.
• Length: The sentence has under 20 characters or over 500 characters (note: this is a bad heuristic for ideographic languages)
• Danger Chars: Over 20% of the characters in the sentence match [0-9{}+/()>]
• Cursedness: The sentence matches a cursed regex (see below)

Cursed Substrings

Based on the initial round of data audits, the authors created a heuristic list of substrings and regexes accounting for a large amount of questionable content. Keep in mind that these all are fed into the pct_questionable score -- a sentence is only excluded from the clean dataset if over 20% of the sentences in that document are flagged as questionable.

notes about cursed substrings:

• low quality sentences ending in the pipe character were very common. Before you ask, this was not Devanagari-script text using a Danda.
• The last few regexes are meant to match A N T S P E A K, List Case, and weirdly regular text (for instance, lists of shipping labels or country codes)

# this implementation is for demonstration and is pretty inefficient;
# to speed it up, use string inclusion (`in`) instead of regex for all but the
# last four, and for those use a compiled regex.
def is_cursed(s):
  return any(re.findall(curse, s) in s for curse in CURSED_SUBSTRINGS)

CURSED_SUBSTRINGS = [" №", "���", "\\|\\s*$", " nr\\.$", "aute irure dolor ", " sunt in culpa qui ", "orem ipsum ", " quis nostrud ", " adipisicing ", " dolore eu ", " cupidatat ", "autem vel eum", "wisi enim ad", " sex ", " porn ", "黄色电影", "mp3", "ownload", "Vol\\.", " Ep\\.", "Episode", " г\\.\\s*$", " кг\\.\\s*$", " шт\\.", "Develop", "Facebook", " crusher ", " xxx ", " ... ... ... ... ... ... ... ... ...", " .... .... .... .... .... .... .... .... ....", " [^ ] [^ ] [^ ] [^ ] [^ ] [^ ] [^ ] [^ ] [^ ]", ", ..,,? ..,,? ..,,? ..,,?"]

Virama Correction

Many languages using Brahmic Abugida (South and Southeast Asian scripts like Devanagari, Khmer, etc.) use some variant on the virama character. For whatever reason, it was found that this character was often messed up in the common crawl snapshots used. Therefore, for the languages bn my pa gu or ta te kn ml si th tl mn lo bo km hi mr ne gom as jv dv bho dz hne ks_Deva mag mni shn yue zh ja kjg mnw ksw rki mtr mwr xnr, a special correction step was done.

For these languages, the authors took the list of all virama characters and removed all unnecessary spaces between each instance of a virama character and the next character with a regex.

'%s' % regex.sub(r' ([%s]) ' % _VIRAMA_CHARS, '\\1', x)

Myanmar Font Compatibility

Prior to 2019, the most popular font for Burmese websites was the Zawgyi font. The authors used Myanmar Tools to convert text.

Several scripts, like the Chinese script, Tibetan script, and Thai, do not use whitespace to separate characters. The languages with this property in this dataset are yue zh ja th lo kjg mnw my shn ksw rki km bo dz.

Alas, the Length aspect of the pct_questionable score was calculated using simplistic whitespace tokenization, and therefore rendered the whole pct_questionable score invalid for those languages. Therefore, for these languages, the "clean" data is identical to the "noisy" data (barring Chinese; see below.)

Special filters

Chinese had a particular issue with pornographic content. After manual inspection a list of strings likely to be present in pornographic content was developed. All pages containing at least one of these strings were removed. Resulted in 17% reduction in number of documents and 56% reduction in file size.

pornsignals = "caoporn caoprom caopron caoporen caoponrn caoponav caopom caoorn 99re dy888 caopro hezyo re99 4438x zooskool xfplay 7tav xxoo xoxo 52av freexx 91chinese anquye cao97 538porm 87fuli 91pron 91porn 26uuu 4438x 182tv kk4444 777me ae86 91av 720lu yy6080 6080yy qqchub paa97 aiai777 yy4480 videossexo 91free 一级特黄大片 偷拍久久国产视频 日本毛片免费视频观看 久久免费热在线精品 高清毛片在线看 日本毛片高清免费视频 一级黄色录像影片 亚洲男人天堂 久久精品视频在线看 自拍区偷拍亚洲视频 亚洲人成视频在线播放 色姑娘综合站 丁香五月啪啪 在线视频成人社区 亚洲人成视频在线播放 久久国产自偷拍 一本道 大香蕉无码 香港经典三级 亚洲成在人线免费视频 天天色综合网 大香蕉伊人久草 欧美一级高清片 天天鲁夜夜啪视频在线 免费黄片视频在线观看 加比勒久久综合 久草热久草在线视频 韩国三级片大全在线观看 青青草在线视频 美国一级毛片 久草在线福利资源 啪啪啪视频在线观看免费 成人福利视频在线观看 婷婷我去也 老司机在线国产 久久成人视频 手机看片福利永久国产 高清国产偷拍在线 大香蕉在线影院 日本高清免费一本视频 男人的天堂东京热 影音先锋男人资源 五月婷婷开心中文字幕 亚洲香蕉视频在线播放 天天啪久久爱视频精品 超碰久久人人摸人人搞".split()

A few more random notes, comparing to common alternative codes for these languages:

• fil for Filipino/Tagalog, not tl
• ak for Twi/Akan, rather than tw. This includes Fante.
• Unfortunately use the macro code chm for Meadow Mari (instead of the correct mhr), and mrj for Hill Mari
• no for Norwegian Bokmål, whereas some resources use nb
• ps for Pashto instead of pbt (Southern Pashto)
• ms for Standard Malay, not zlm
• sq for Albanian, and don't distinguish dialects like Gheg (aln) and Tosk (als)
• ber as the code for Tamazight, after consultation with Tamazight speakers opining that the dialect distinctions are not significant. Other resources use the individual codes like tzm and kab.
• Macrocode qu for Quechua. In practice, this seems usually to be a mix of the Ayacucho and Cusco dialects. Other resources, like NLLB, may use the dialect code, e.g. quy for Ayacucho Chanka. The same is true for a few other macro codes, like ff (Macro code for Fulfulde, whereas other sources may use e.g. fuv.)
• Really, there are notes that can be made about almost any code, from the well-accepted conventions like zh for Mandarin, to many dialectical notes, like which variant of Hmong really is the hmn data? But the above ones are made specifically for ones where the authors are aware of other datasources floating out there that use different conventions.

Audit

Following Quality at a Glance, the authors performed an "audit" of every corpus in this dataset. Although the authors did not speak most languages, they were able to give high-level comments on the general quality. They looked at a sample of 20 documents of each language.

After an initial round of auditing, they devised a new set of filters and applied them. They then re-did all audits.

Overall notes from the audit

The decision was to include languages that looked noisy, but omit any language that was clearly majority noise, or only had 20 or fewer docs. This is a low bar -- twenty documents can be very little indeed, and some of the corpora released are quite noisy, but all of them should have at least the potential to be used in some useful way. The motivation for not releasing nonsense or tiny datasets is to not give a false sense of how multilingual this dataset actually is ("Representation washing"), as recommended by Quality at a Glance.

A few overarching points:

• Many low-resource languages only had Bible text, or in some cases jw.org data. These are marked in the rows below. Generally ok bible means that 100% of the audited sentences were Biblical, whereas if bible is simply mentioned in the note, it was not the only source of data.
• Indian languages in the Latin script had a high concentration of pornographic content.

Renames and Merges as a result of the Audit

In several cases, it was clear from the audit that the corpora were not in the languages that the LangID model claimed they were. This led to the following renames:

• dty renamed to zxx-xx-dtynoise, aka a "language" of noise. This is mainly mis-rendered PDFs and may have some practical applications for decoding said.
• fan renamed to bum
• ss-SZ renamed to ss -- this was just a result of us having inconsistent data labels.
• cjk merged into the gil dataset
• bjj merged into the awa dataset

Canaries

Canaries are provided in separate canaries folder. Canaries are organized into three directions: monolingual hosts canaries designed for the MADLAD-400 monody data, multiway for the multiway data, and generic the generic canaries generated only from the model's vocabulary.

• Monolingual: Canaries here are organized by the language the canary was generated from. This corresponds exactly to the translate_copy setting in the paper, where the source and target language match.

• Multiway: Canaries here are organized in one of two fashions. to_XX indicates canaries organized by the target language (and where the source language could be any language). XX-XX indicates the canaries (interleaved_both and interleaved_mislabeled_both) designed for a specific pair of languages.

Within each subdirectory above, canaries are into separate files named by the canary type. There is always only a single file for each canary type. The generic folder contains within it the four canary types.

Canaries can be mixed in with normal training data to then be analyzed post-hoc to training

References

Raffel, Colin, et al. "Exploring the limits of transfer learning with a unified text-to-text transformer." J. Mach. Learn. Res. 21.140 (2020): 1-67.

Contact

Please reach out to {snehakudugunta, icaswell}꩜google.com. For questions about the canaries, reach out to cchoquette@google.com

License

This data is released with the CC-BY-4.0 license.

Detailed notes from the audit

Here are the notes on all languages, along with the number of documents found, and the final decision made with respect to including the language in this dataset.

A few comments too long to fit in the table above:

• alt: WAIT THIS IS AMAZING IT IS ACTUALLY ALTAI! e.g. from urls like https://altaicholmon.ru/2020/02/28/jarashty-la-jajaltany-jarkyndu-lekeri/
• tly-IR: They all look like boilerplate content, e.g., list of keywords/search queries used to bump page ranking in search results. Not any useful material for translation. Remove.
• zap: pls note that at least some Zapotec speakers tend to view it as one language, not as a million dialects like ISO does. However, some are certainly mutually unintelligible, complicating the matter.
• zh-Latn: The biggest problem is that several examples are not in Latin Chinese (i.e., romanization in my understanding) but in English or mixed English and Chinese. For those data in Latin Chinese, their quality seems to be good.
• zh: Many examples are porn-related, particularly those very long documents. Also, there are some examples of traditional Chinese.

Final Dataset information

The number of documents, sentences, tokens, characters, and bytes for the noisy and clean splits of the data. Note that the "toks" field below uses whitespace for tokenization, so is not appropriate for non-whitespace-separating languages like Chinese (see section above). Note that the english subset in this version is missing 18% of documents that were included in the published analysis of the dataset. These documents will be incoporated in an update coming soon.

Citation Information

@misc{kudugunta2023madlad400,
      title={MADLAD-400: A Multilingual And Document-Level Large Audited Dataset}, 
      author={Sneha Kudugunta and Isaac Caswell and Biao Zhang and Xavier Garcia and Christopher A. Choquette-Choo and Katherine Lee and Derrick Xin and Aditya Kusupati and Romi Stella and Ankur Bapna and Orhan Firat},
      year={2023},
      eprint={2309.04662},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}