Pile 13B tokens: filtered

Filtered half of a token-matched pair; the other half is pile_unfiltered_13B.

Built from shards 0-269 of EleutherAI/the_pile_deduplicated for a controlled comparison: the same GPT-2-small model trained on filtered vs. unfiltered data.

Token-matched by construction. For every source shard i:

• pile_filtered_13B holds the documents kept by the filtering pipeline for shard i;
• pile_unfiltered_13B holds a random sample of raw documents from the same shard i (seed = i), drawn until it has exactly the same number of GPT-2 tokens (the last document is truncated).

That is 103.7% of the CS336 assignment 4 training budget (100k steps x 128 batch x 2 GPUs x 512 context = 13,107,200,000 tokens), so a full run sees no repeated data.

Filtering pipeline (language ID, PII masking, Gopher quality rules, fastText quality classifier): 251 shards; 19 early shards additionally had NSFW/toxicity filtering. Each shard's manifest records which stages ran (filtered.pipeline_columns). No additional deduplication beyond the source dataset's own.

Layout

• tokens/train-XXXXX.bin: uint16 GPT-2 token ids, <|endoftext|> (50256) after every document
• text/train-XXXXX.parquet: the documents
• manifests/train-XXXXX.json: doc/token counts, seed, source file

Train on it

huggingface-cli download ViratGarg/pile_filtered_13B --repo-type dataset --include "tokens/*" --local-dir data
cat data/tokens/train-*.bin > train.bin   # then set paths.train_bin: train.bin