The best fully open-source L language models with the entire pipeline, for training code, data, and more! .

OLMo 2 (Nov. 2024)

• Collection
• 7B base model
• 13B base model
• 7B instruct
• 13B instruct
• Annealing dataset
• Training Code (1st gen.)
• Training Code (2nd gen.)
• Post-train Code
• Eval Code
• Data Processing Toolkit
• Demo

AMD OLMo (Oct. 2024)

• 1B model
  • SFT
  • SFT-DPO
• Blog post

HuggingFace SmolLM (v2 Oct. 2024)

• SmolLM 2 collection
• SmolLM 2 pretraining data: TBD
• SmolLM instruction mix
• SmolLM collection
• SmolLM pretraining data
• Synthetic pretrain corpus
• Fineweb pretrain corpus
  • Edu Subset
  • Fineweb 2 (multilingual)
• SmolLM repo
  • Training Code
  • Eval Code
  • Data Processing Code
• Blogposts:
  • Fineweb
  • Fineweb v2
  • SmolLM

DataComp (Jun. 2024)

• 1B Model
  • Instruct
• 7B Model
  • Instruct
  • 8K Extension
• Pretrain Dataset
• Training + Eval Code
• Paper
• Evalchemy (Post-training Eval Code)

Databricks / formerly Mosaic ML

• Streaming Datasets
• Eval Gauntlet
• Pretraining Code
  • Megablocks (MoE Training)
• DBRX Collection

LLM 360

• Analysis360
• K2-65B:
  • Collection
  • Data
  • Data Preparation Code
  • Training Code
• CrystalCoder-7B:
  • Data
  • Data Preparation Code
  • Training Code
• Amber-7B:
  • Collection
  • Data
  • Data Preparation Code
  • Training Code
• Paper

EleutherAI

• Pythia

  • Models Collection
  • Central Repo
  • Paper

• GPT-NeoX-20B

  • Model
  • Paper

• Llema-7B

  • Central Repo
  • Paper
  • Dataset
  • Blogpost

• Training Code

• Eval Code

  • Paper

• The Pile Dataset

  • Paper

Cerebras

• Cerebras-GPT
  • Models Collection
  • Paper
  • Blogpost
  • Training Code

RWKV

• RWKV-5 Models Collection
• RWKV-6 Models Collection
• Training Code
• Papers:
  • RWKV-4
  • RWKV-5 and RWKV-6

M.A.P.

• Models Collection
• Datasets
• Paper
• Code

Zyphra

• Zamba 2 Models:
  • 7B
    • 7B Instruct
    • Blogpost
  • 2.7B
    • Instruct
    • Blogpost
  • 1.2B
    • Instruct
    • Blogpost
  • Paper
• Zyda 2 Dataset
  • Blogpost
  • Paper

Together.AI

• RedPajama v1 Dataset
• RedPajama v2 Dataset
• Data Code

NVIDIA

• Pretraining Code
  • Megatron-LM
  • NeMo
• Data Processing
• Post-training Code

PyTorch / Meta

• Torch Titan Pretraining Code
• Meta Lingua Pretraining Code

Open-sci, OpenEuroLLM, LAION AI (Sept. 2025)

• Paper
• Blog
• HF collections

LLäMmlein (German LLaMa)

• Paper
• Blog
• 120M, 1B, 7B, with intermediate checkpoints
• Dataset
• Pretraining codebase