Unfiltered half of a token-matched pair; the other half is pile_filtered_13B.
Built from shards 0-269 of EleutherAI/the_pile_deduplicated for a controlled comparison: the same GPT-2-small model trained on filtered vs. unfiltered data.
Token-matched by construction. For every source shard i:
pile_filtered_13B holds the documents kept by the filtering pipeline for shard i;pile_unfiltered_13B holds a random sample of raw documents from the same shard i (seed = i),
drawn until it has exactly the same number of GPT-2 tokens (the last document is truncated).| filtered | unfiltered | |
|---|---|---|
| shards | 270 | 270 |
| tokens | 13,589,978,522 | 13,589,978,522 |
| documents | 11,023,784 | 8,374,559 |
That is 103.7% of the CS336 assignment 4 training budget (100k steps x 128 batch x 2 GPUs x 512 context = 13,107,200,000 tokens), so a full run sees no repeated data.
Filtering pipeline (language ID, PII masking, Gopher quality rules, fastText quality classifier):
251 shards; 19 early shards
additionally had NSFW/toxicity filtering. Each shard's manifest records which stages ran (filtered.pipeline_columns).
No additional deduplication beyond the source dataset's own.
tokens/train-XXXXX.bin: uint16 GPT-2 token ids, <|endoftext|> (50256) after every documenttext/train-XXXXX.parquet: the documentsmanifests/train-XXXXX.json: doc/token counts, seed, source filehuggingface-cli download ViratGarg/pile_unfiltered_13B --repo-type dataset --include "tokens/*" --local-dir data
cat data/tokens/train-*.bin > train.bin # then set paths.train_bin: train.bin
Unfiltered half of a token-matched pair; the other half is pile_filtered_13B.
Built from shards 0-269 of EleutherAI/the_pile_deduplicated for a controlled comparison: the same GPT-2-small model trained on filtered vs. unfiltered data.
Token-matched by construction. For every source shard i:
pile_filtered_13B holds the documents kept by the filtering pipeline for shard i;pile_unfiltered_13B holds a random sample of raw documents from the same shard i (seed = i),
drawn until it has exactly the same number of GPT-2 tokens (the last document is truncated).| filtered | unfiltered | |
|---|---|---|
| shards | 270 | 270 |
| tokens | 13,589,978,522 | 13,589,978,522 |
| documents | 11,023,784 | 8,374,559 |
That is 103.7% of the CS336 assignment 4 training budget (100k steps x 128 batch x 2 GPUs x 512 context = 13,107,200,000 tokens), so a full run sees no repeated data.
Filtering pipeline (language ID, PII masking, Gopher quality rules, fastText quality classifier):
251 shards; 19 early shards
additionally had NSFW/toxicity filtering. Each shard's manifest records which stages ran (filtered.pipeline_columns).
No additional deduplication beyond the source dataset's own.
tokens/train-XXXXX.bin: uint16 GPT-2 token ids, <|endoftext|> (50256) after every documenttext/train-XXXXX.parquet: the documentsmanifests/train-XXXXX.json: doc/token counts, seed, source filehuggingface-cli download ViratGarg/pile_unfiltered_13B --repo-type dataset --include "tokens/*" --local-dir data
cat data/tokens/train-*.bin > train.bin # then set paths.train_bin: train.bin