LLM Pretraining Datasets & Corpora
10 repos
Large-scale web-derived datasets and corpus curation pipelines designed for training large language models. These repositories focus on collecting, filtering, deduplicating, and preprocessing raw web text at massive scale—work that underpins modern LLM pretraining. Central projects like FineWeb, Dolma, and Ultra-FineWeb represent different approaches to dataset construction, with specialized variants for educational content and refined quality filtering.
llm ·1,985
language-modeling ·1,075
casual-lm ·1,075
web-corpus ·910
pretraining ·910
data-filtering ·628
high-quality ·437
function-calling ·282
data-refinement ·282
programmatic-editing ·282
