
This is a synthetic dataset designed for supervised finetuning (SFT) of LLMs. It was used to build SmolLM2-Instruct family of models and contains 1M samples. More details in our paper https://arxiv.org/abs/2502.02737
During the development of SmolLM2, we observed that models finetuned on public SFT datasets underperformed compared to other models with proprietary instruction datasets. To address this gap, we created new synthetic datasets that improve instruction following while covering diverse tasks including text editing, rewriting, summarization, and reasoning. Through a series of data ablations at 1.7B scale, we enhanced our SFT mix by incorporating public datasets to strengthen specific capabilities such as mathematics, coding, system prompt following and long-context understanding.
All the new datasets were generated with distilabel and you can find the generation code here https://github.com/huggingface/smollm/tree/main/text/data/smoltalk.
You can load a dataset using
from datasets import load_dataset
ds = load_dataset("HuggingFaceTB/smoltalk", "all", split="train")
# to load the train split of a specific subset such as smol-magpie-ultra, you can do
ds = load_dataset("HuggingFaceTB/smoltalk", "smol-magpie-ultra", split="train")
The mix consists of:
New datasets
Existing public datasets To enhance capabilities in mathematics, coding, system prompts, and long-context understanding, we fine-tuned SmolLM2-1.7B on various public SFT datasets and included subsets of the best performing ones using tuned ratios. These include:
You can find the code for generating the new datasets with distilabel here: https://github.com/huggingface/smollm. The ablation details will be included in an upcoming blog post.
All the new datasets (Smol-Magpie-Ultra, Smol-contraints, Smol-rewrite, Smol-summarize) are licensed under Apache 2.0. For the existing public datasets, please refer to the original dataset for the license Dataset composition
We compare SmolTalk to the recent Orca AgentInstruct 1M dataset by finetuning SmolLM2 on both datasets using the same training setup (we train for 2 epochs, using a learning rate of 3e-04, a sequence length of 8192 and a global batch size of 16).

We also observe significant improvements at 7B scale when fine-tuning Mistral-7B on SmolTalk, notably on IFEval, BBH, GS8Mk and MATH.

For SmolLM2-135M-Instruct and SmolLM2-360M-Instruct, we use a subset of the dataset that is more suitable for these smaller models. For instance, we only include samples from Smol-Magpie-Ultra with more concise conversations and exclude advanced math datasets. You can find the dataset here: https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk
The training code is available here https://github.com/huggingface/alignment-handbook/tree/main/recipes/smollm2
@misc{allal2025smollm2smolgoesbig,
title={SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model},
author={Loubna Ben Allal and Anton Lozhkov and Elie Bakouch and Gabriel Martín Blázquez and Guilherme Penedo and Lewis Tunstall and Andrés Marafioti and Hynek Kydlíček and Agustín Piqueres Lajarín and Vaibhav Srivastav and Joshua Lochner and Caleb Fahlgren and Xuan-Son Nguyen and Clémentine Fourrier and Ben Burtenshaw and Hugo Larcher and Haojun Zhao and Cyril Zakka and Mathieu Morlon and Colin Raffel and Leandro von Werra and Thomas Wolf},
year={2025},
eprint={2502.02737},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2502.02737},
}

This is a synthetic dataset designed for supervised finetuning (SFT) of LLMs. It was used to build SmolLM2-Instruct family of models and contains 1M samples. More details in our paper https://arxiv.org/abs/2502.02737
During the development of SmolLM2, we observed that models finetuned on public SFT datasets underperformed compared to other models with proprietary instruction datasets. To address this gap, we created new synthetic datasets that improve instruction following while covering diverse tasks including text editing, rewriting, summarization, and reasoning. Through a series of data ablations at 1.7B scale, we enhanced our SFT mix by incorporating public datasets to strengthen specific capabilities such as mathematics, coding, system prompt following and long-context understanding.
All the new datasets were generated with distilabel and you can find the generation code here https://github.com/huggingface/smollm/tree/main/text/data/smoltalk.
You can load a dataset using
from datasets import load_dataset
ds = load_dataset("HuggingFaceTB/smoltalk", "all", split="train")
# to load the train split of a specific subset such as smol-magpie-ultra, you can do
ds = load_dataset("HuggingFaceTB/smoltalk", "smol-magpie-ultra", split="train")
The mix consists of:
New datasets
Existing public datasets To enhance capabilities in mathematics, coding, system prompts, and long-context understanding, we fine-tuned SmolLM2-1.7B on various public SFT datasets and included subsets of the best performing ones using tuned ratios. These include:
You can find the code for generating the new datasets with distilabel here: https://github.com/huggingface/smollm. The ablation details will be included in an upcoming blog post.
All the new datasets (Smol-Magpie-Ultra, Smol-contraints, Smol-rewrite, Smol-summarize) are licensed under Apache 2.0. For the existing public datasets, please refer to the original dataset for the license Dataset composition
We compare SmolTalk to the recent Orca AgentInstruct 1M dataset by finetuning SmolLM2 on both datasets using the same training setup (we train for 2 epochs, using a learning rate of 3e-04, a sequence length of 8192 and a global batch size of 16).

We also observe significant improvements at 7B scale when fine-tuning Mistral-7B on SmolTalk, notably on IFEval, BBH, GS8Mk and MATH.

For SmolLM2-135M-Instruct and SmolLM2-360M-Instruct, we use a subset of the dataset that is more suitable for these smaller models. For instance, we only include samples from Smol-Magpie-Ultra with more concise conversations and exclude advanced math datasets. You can find the dataset here: https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk
The training code is available here https://github.com/huggingface/alignment-handbook/tree/main/recipes/smollm2
@misc{allal2025smollm2smolgoesbig,
title={SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model},
author={Loubna Ben Allal and Anton Lozhkov and Elie Bakouch and Gabriel Martín Blázquez and Guilherme Penedo and Lewis Tunstall and Andrés Marafioti and Hynek Kydlíček and Agustín Piqueres Lajarín and Vaibhav Srivastav and Joshua Lochner and Caleb Fahlgren and Xuan-Son Nguyen and Clémentine Fourrier and Ben Burtenshaw and Hugo Larcher and Haojun Zhao and Cyril Zakka and Mathieu Morlon and Colin Raffel and Leandro von Werra and Thomas Wolf},
year={2025},
eprint={2502.02737},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2502.02737},
}