Data released with ReScraper, a 0.6B refiner that replaces the heuristic HTML-to-text stack of a pretraining data pipeline with a single small language model.
cx-cmu/ReScrapercxcscmu/ReScraperAll four subsets come from the same source pool: the 10% raw-HTML sample of the DCLM pool
(dclm-pool-400m-1x-html-jsonl-step3a-10pct), 18,025,558 pages over 10,319 shards.
| Path | Rows | Size | What it is |
|---|---|---|---|
data/ReScraper-Corpus/ | 10,056,105 | 12.4 GB | The curated pretraining corpus: model output after the post-filter and Bloom-filter deduplication. 7.44B GPT-NeoX-20B tokens. |
data/ReScraper-Pool-Output/ | 10,646,236 | 13.3 GB | The refiner's raw output over the whole pool, before deduplication. Use this to study the model's behaviour page by page. |
data/ReScraper-SFT-Stage1/ | 1,383,115 | 3.9 GB | Stage 1 training set: extraction plus the operation decision. |
data/ReScraper-SFT-Stage2/ | 131,484 | 0.4 GB | Stage 2 training set, which upweights <rewrite> so the model learns that operation. |
Files are Parquet, split into roughly 1 GB parts named
<subset>-en-part-XXXX-of-YYYY.parquet.
Corpus and pool output
| Field | Type | Meaning |
|---|---|---|
text | string | the cleaned page |
operation | string | the operation the model chose: <keep>, <edit> or <rewrite> (deleted pages are not materialized) |
bff_contained_ngram_count_before_dedupe | int | deduplication bookkeeping (corpus only) |
SFT sets
| Field | Type | Meaning |
|---|---|---|
input | string | the rendered page, one block per line with <lid:n> line ids |
output | string | the target program: an <extract> block of line removals followed by the operation tag |
from datasets import load_dataset
corpus = load_dataset("cx-cmu/ReScraper-Data", data_dir="data/ReScraper-Corpus", split="train")
sft = load_dataset("cx-cmu/ReScraper-Data", data_dir="data/ReScraper-SFT-Stage1", split="train")
To reproduce pretraining, tokenize the corpus with the GPT-NeoX-20B tokenizer; the code repository has the tokenization and pretraining scripts.
Apache 2.0. The underlying pages come from Common Crawl via the DCLM pool and keep their original terms.
Data released with ReScraper, a 0.6B refiner that replaces the heuristic HTML-to-text stack of a pretraining data pipeline with a single small language model.
cx-cmu/ReScrapercxcscmu/ReScraperAll four subsets come from the same source pool: the 10% raw-HTML sample of the DCLM pool
(dclm-pool-400m-1x-html-jsonl-step3a-10pct), 18,025,558 pages over 10,319 shards.
| Path | Rows | Size | What it is |
|---|---|---|---|
data/ReScraper-Corpus/ | 10,056,105 | 12.4 GB | The curated pretraining corpus: model output after the post-filter and Bloom-filter deduplication. 7.44B GPT-NeoX-20B tokens. |
data/ReScraper-Pool-Output/ | 10,646,236 | 13.3 GB | The refiner's raw output over the whole pool, before deduplication. Use this to study the model's behaviour page by page. |
data/ReScraper-SFT-Stage1/ | 1,383,115 | 3.9 GB | Stage 1 training set: extraction plus the operation decision. |
data/ReScraper-SFT-Stage2/ | 131,484 | 0.4 GB | Stage 2 training set, which upweights <rewrite> so the model learns that operation. |
Files are Parquet, split into roughly 1 GB parts named
<subset>-en-part-XXXX-of-YYYY.parquet.
Corpus and pool output
| Field | Type | Meaning |
|---|---|---|
text | string | the cleaned page |
operation | string | the operation the model chose: <keep>, <edit> or <rewrite> (deleted pages are not materialized) |
bff_contained_ngram_count_before_dedupe | int | deduplication bookkeeping (corpus only) |
SFT sets
| Field | Type | Meaning |
|---|---|---|
input | string | the rendered page, one block per line with <lid:n> line ids |
output | string | the target program: an <extract> block of line removals followed by the operation tag |
from datasets import load_dataset
corpus = load_dataset("cx-cmu/ReScraper-Data", data_dir="data/ReScraper-Corpus", split="train")
sft = load_dataset("cx-cmu/ReScraper-Data", data_dir="data/ReScraper-SFT-Stage1", split="train")
To reproduce pretraining, tokenize the corpus with the GPT-NeoX-20B tokenizer; the code repository has the tokenization and pretraining scripts.
Apache 2.0. The underlying pages come from Common Crawl via the DCLM pool and keep their original terms.