TxT360: A Top-Quality LLM Pre-training Dataset Requires the Perfect Blend

Changelog

Details of v1.1 Additions

• TxT360_BestOfWeb: This is a filtered version of the TxT360 dataset, created using the ProX document filtering model. The model is similar to the FineWeb-Edu classifier, but also assigns an additional format score that considers how a document is formatted.

• TxT360_QA: Synthetic QA pairs generated for each document using Mistral-7B-Instruct-v0.3. QA pairs are appended to the end of every document in the format:

  {
    "text": "{ORIGINAL_DOCUMENT_TEXT}\n\nQ: {QUESTION_1}\nA: {ANSWER_1}\n\nQ: {QUESTION_2}\nA: {ANSWER_2}......{ANSWER_N}\n",
    "meta": {original TxT360 meta}
  }
  

  The number of QA pairs may differ for each document, providing diverse question-answering supervision.

• europarl-aligned: Europarl v7 data processed to align English source text with parallel corpora in multiple languages. Each sample concatenates the same content in different languages. Steps include reading English source text, matching with parallel corpus data, and concatenating multilingual content for robust cross-lingual training without any order., e.g.:

  {
    "text": "# English\n\n[English content]\n\n# French\n\n[French content]\n\n # Italian\n\n [Italian content]\n...",
    "meta": {
                "language":"fi-de-nl-el-it-fr-en-pt-sv-da-es",
                "src_file":"ep-00-01-17"
            }
  }
  

• wikipedia_extended: An enhanced version of Wikipedia data that:

  • Appends abstracts of outgoing linked articles from the source article's abstract to each Wikipedia document.
  • Creates a contextually dense document with interconnected information from related articles.
  • Enables long-context training by allowing models to process extended sequences of linked content.
  • Enhances model ability to understand topic relationships, maintain coherence in long contexts, and generate accurate responses across topics.

{
  "text": "{ORIGINAL ARTICLE}\n\n{First Link Title}\n\"{First Link Abstract}\"\n\n{Second Link Title}\n\"{Second Link Abstract}\"..."
}

We introduce TxT360 (Trillion eXtracted Text) the first dataset to globally deduplicate 99 CommonCrawl snapshots and 14 commonly used non-web data sources (e.g. FreeLaw, PG-19, etc.) providing pretraining teams with a recipe to easily adjust data weighting, obtain the largest high-quality open source dataset, and train the most performant models.

TxT360 Compared to Common Pretraining Datasets

• TxT360 does not include code. This decision was made due to the perceived low duplication code with other sources.

Complete details on the dataset can be found in our blog post here.

TxT360 Performance

To evaluate the training efficiency of our dataset, we sampled 1.5T tokens from both FineWeb and TxT360 (using the aforementioned weighting) and conducted a training ablation on an 8x8B Mixture-of-Experts architecture, similar to Mixtral. We compared the learning curves by tracking training loss, validation scores, and performance across a wide array of diverse evaluation benchmarks. The validation set was sampled independently from SlimPajama. Note that this experiment is done on a slightly earlier version of the dataset.

Initial Data Representation

To produce TxT360, a comprehensive data processing pipeline was designed to account for the nuances of both web and curated datasets. The pipeline presents a unified framework for processing both data types, making it convenient and easily adaptive for users to revise and fine-tune the pipeline for their own use cases.

Web datasets are inherently noisy and varied. The TxT360 pipeline implements sophisticated filtering and deduplication techniques to clean and remove redundancies while preserving data integrity.

Curated datasets are typically structured and consistently formatted, but also can cause troubles with their own special formatting preferences. TxT360 filters these sources with selective steps to maintain their integrity while providing seamless integration into the larger dataset. Both data source types are globally deduplicated together resulting in ~5T tokens of high-quality data. The table below shows the source distribution of TxT360 tokens.

We further highlight the importance of mixing the datasets together with the right blend. The raw distribution of the deduplicated dataset is actually suboptimal, a simple working recipe is provided in the studies section. This recipe will create a dataset of 15T+ tokens, the largest high quality open source pre-training dataset.

The TxT360 blog post provides all the details behind how we approached and implemented the following features:

CommonCrawl Data Filtering

Complete discussion on how 99 Common Crawl snapshots were filtered and comparison to previous filtering techinques (e.g. Dolma, DataTrove, RedPajamaV2).

Curated Source Filtering

Each data source was filtered individually with respect to the underlying data. Full details and discussion on how each source was filter are covered.

Global Deduplication

After the web and curated sources were filtered, all sources globally deduplicated to create TxT360. The tips and tricks behind the deduplication process are included.

Dataset Structure

The dataset is organized under the data directory, with each subdirectory representing a data subset. Below is an overview of the structure and organization of these subsets:

├── data
    ├── common-crawl  # data subset
        ├── CC-MAIN-2013-20  # common-crawl dumps
            ├── 1-1  # number of duplicates
                ├── chunk_000_0000.jsonl.gz
                ├── ...
            ├── 2-5
                ├── chunk_000_0000.jsonl.gz
                ├── ...
            ├── ...
        ├── CC-MAIN-2013-48
            ├── 1-1
                ├── chunk_000_0000.jsonl.gz
                ├── ...
            ├── ...
        ├── ...
    ├── dm_math
        ├── full_data_1
            ├── 0_11255.jsonl
            ├── ...
        ├── full_data_2
            ├── 10000_11255.jsonl
            ├── ...
    ├── arxiv
        ├── 1-1  # number of duplicates
            ├── 0_171.jsonl
            ├── ...
        ├── 2-5
            ├── 0_2.jsonl
            ├── ...
        ├── ...
    ├── europarl
        ├── 1-1  # number of duplicates
            ├── 0_6.jsonl
            ├── ...
        ├── 2-5
            ├── 0_0.jsonl
            ├── ...
        ├── ...
    ├── ...

Common Crawl (common-crawl)

Each subdirectory under common-crawl corresponds to a specific dump of the dataset. Inside each dump folder, the data is further segmented into buckets based on the number of duplicates identified during deduplication:

• 1-1: Contains documents with no duplicates across the dataset.
• 2-5, 6-10, 11-100, 101-1000, 1001-30000000: Each contains documents that fall within the respective range of duplicates.

Example path: data/common-crawl/CC-MAIN-2013-20/1-1/chunk_000_0000.jsonl.gz

DM Math (dm_math)

The dm_math subset is divided into two subfolders to comply with the limit of 10,000 files per folder in a HuggingFace Repository:

Example path: data/dm_math/full_data_1/0_11255.jsonl

Others

Similar to common-crawl, other curated data subsets, such as arxiv, europal, etc., are organized by the number of duplicates:

• 1-1, 2-5, 6-10, 11-100, 101-1000, 1001-inf

Kindly note that some data subsets might not include the folder 1001-inf (1001-30000000 in common-crawl) or might contain only a few documents in such a folder due to the rarity of documents duplicated more than 1000 times.

Data Schema

Common Crawl (common-crawl)

The documents in common-crawl follow the schema:

{'text': '...',  # texts in the document
 'meta': 
    {
        'lang': 'en',  # top 1 language detected by fastText model
        'lang_score': 0.912118136882782,  # language score for the detected language
        'url': 'http://www.shopgirljen.com/2017/10/lg-celebrates-5-years-of-lg-oled-tv.html',  # the url that raw webpage is scraped from
        'timestamp': '2024-07-24T00:56:12Z',  # timestamp from Common Crawl raw data
        'cc-path': 'crawl-data/CC-MAIN-2024-30/segments/1720763518130.6/warc/CC-MAIN-20240723224601-20240724014601-00300.warc.gz',  # the path of the document in the raw Common Crawl
        'quality_signals':
            {
                'url_score': 0.0,
                'fraction_of_duplicate_lines': 0.0,
                'fraction_of_characters_in_duplicate_lines': 0.0,
                'fraction_of_duplicate_paragraphs': 0.0,
                'fraction_of_characters_in_duplicate_paragraphs': 0.0,
                'fraction_of_characters_in_most_common_ngram': [[2, 0.03626373626373627],
                    [3, 0.03296703296703297],
                    [4, 0.01868131868131868]],
                'fraction_of_characters_in_duplicate_ngrams': [[5, 0.01868131868131868],
                    [6, 0.01868131868131868],
                    [7, 0.01868131868131868],
                    [8, 0.0],
                    [9, 0.0],
                    [10, 0.0]],
                'fraction_of_words_corrected_in_lines': 0.0,
                'fraction_of_lines_ending_with_ellipsis': 0.0,
                'fraction_of_lines_starting_with_bullet_point': 0.0,
                'fraction_of_lines_with_toxic_words': 0.0,
                'num_of_lines_with_toxic_words': 0,
                'num_of_toxic_words': 0,
                'word_count': 358,
                'mean_word_length': 5.083798882681564,
                'num_of_sentences': 19,
                'symbol_to_word_ratio': 0.0,
                'fraction_of_words_with_alpha_character': 1.0,
                'num_of_stop_words': 82,
                'num_of_paragraphs': 0,
                'has_curly_bracket': False,
                'has_lorem_ipsum': False,
                'orig_text_has_dup_lines': False
                },
        'dup_signals': 
            {
                'dup_doc_count': 166,  # the number of duplicated documents
                'dup_dump_count': 57,  # the number of dumps that the duplicated documents are from
                'dup_details':   # the dump distribution of the duplicated documents
                    {
                        '2024-30': 2,
                        '2024-26': 1,
                        '2024-22': 1,
                        ...
                        }
                }
        },
 'subset': 'commoncrawl'}

Please note that documents without duplicates, located in folders */1-1/, have an empty dup_signals field. Additionally, some documents with duplicates might include an unknown entry within the dup_details. One example could be:

{'text': '...',  # texts in the document
 'meta': 
    {
        ...
        'dup_signals': 
            {
                'dup_doc_count': 7,
                'dup_dump_count': 3,
                'dup_details':
                    {
                        'unknown': 4,
                        '2024-30': 1,
                        '2024-26': 1,
                        '2024-22': 1,
                        }
                }
        },
 'subset': 'commoncrawl'}

This occurs because the distribution of duplicates across dumps was not recorded in the early stages of our deduplication process, and only the total count of duplicate documents (dup_doc_count) was maintained. Due to the high cost of rerunning the deduplication, we have opted to label these distributions as unknown when integrating them with other documents for which duplicate distribution data is available. In these cases, the dup_dump_count is calculated excluding the unknown.

Citation

BibTeX:

@misc{txt360data2024,
      title={TxT360: A Top-Quality LLM Pre-training Dataset Requires the Perfect Blend}, 
      author={Liping Tang, Nikhil Ranjan, Omkar Pangarkar, Xuezhi Liang, Zhen Wang, Li An, Bhaskar Rao, Linghao Jin, Huijuan Wang, Zhoujun Cheng, Suqi Sun, Cun Mu, Victor Miller, Xuezhe Ma, Yue Peng, Zhengzhong Liu, Eric P. Xing},
      year={2024}
}