SmolTalk2

Dataset description

This dataset contains three subsets (Mid, SFT, Preference) that correspond to the three phases of Post-Training for SmolLM3-3B. You can find more details in our blog post about how we used the data in each of the stages SmolLM3.

The specific weight of each subset is available in the training recipe in SmolLM's repository.

You can load a dataset using

from datasets import load_dataset

# To load the train split of a specific subset, such as Mixture-of-Thoughts, you can do
ds = load_dataset("HuggingFaceTB/smoltalk2", "SFT", split=["Mixture-of-Thoughts_science", "table_gpt_no_think"])

Dataset Composition

Mid-Training (Mid)

The mid-training dataset has a total of 4.8M rows and is composed of 2 datasets that we decontaminate to remove samples present in the benchmarks used for evaluation.

The datasets are:

• Llama-Nemotron-Post-Training-Dataset: 3.64M rows
• OpenThoughts3-1.2M: 1.14M rows.

SFT (SFT)

The total mix consists of 25 datasets, which we decontaminated to remove samples present in the benchmarks used for evaluation and remove samples containing emojis. We also created the chat_template_kwargs column by extracting any system message or tool descriptions already present in the dataset.

We make a distinction between datasets with and without reasoning traces, denoted by the suffixes think and no_think, respectively. The 10 think datasets have a total of 1.5M rows, and the 15 no_think datasets have a total of 1.9M rows.

The think datasets are:

• SmolTalk (multilingual-8languages): 244736 rows generated with Qwen3-32B with the prompts in SmolTalk.
• SmolTalk (everyday-conversations): 244736 rows generated with Qwen3-32B with the prompts in SmolTalk.
• SmolTalk (systemchats-30k): 244736 rows generated with Qwen3-32B with the prompts in SmolTalk.
• LongAlign-64k-context-lang-annotated: 7526 rows generated with Qwen3-32B with the prompts in LongAlign-64k.
• [NEW] smolagents-toolcalling-traces: 9079 rows.
  • We generate tool calling data with reasoning traces using deepseek-ai/DeepSeek-V3-0324.
• [NEW] Multi-Turn IF: 28217 rows.
  • We follow Multi-IF's approach to generate multi turn data. We source prompts from Tulu 3 Personas IF, generate 2 verifiable turns using Qwen3-235B-A22B, and generate responses with Qwen3-32B in reasoning mode.
• s1k-1.1: 835 rows.
• OpenThoughts3-1.2M: 1133524 rows.
• Aya: 15222 rows generated with Qwen3-32B with the prompts in Aya.
• Table-GPT: 13201 rows generated with Qwen3-32B with the prompts in Table-GPT.

The no_think datasets are:

• [NEW] SmolTalk (multilingual-8languages): 254047 rows.
  • Following Qwen 2.5 report, we first translate the prompts in Smol-Magpie-Ultra and Smol-Constraints using Qwen to 8 languages (fr, es, it, pt, de, ar, ru, zh) while respecting local conventions (units, currency, etc.). We then use the model to generate answers for each translated instruction in the target language.
• SmolTalk (everyday-conversations): 2260 rows.
• SmolTalk (systemchats-30k): 33997 rows.
• SmolTalk (smollm3_smol-magpie-ultra): 406843 rows.
• SmolTalk (smollm3_explore-instruct-rewriting): 30391 rows.
• SmolTalk (smollm3_smol-rewrite): 53262 rows.
• SmolTalk smollm3_smol-summarize: 96061 rows.
• Mixture of Thoughts (science): 86110 rows where we remove the reasoning trace.
• Tulu 3 SFT Personas IF: 29970 rows.
• hermes-function-calling-v1: 8961 rows.
• Table-GPT: 13203 rows.
• OpenHermes-2.5: 384900 rows.
• OpenThoughts3-1.2M: 435193 rows where we remove the reasoning trace.
• LongAlign-64k-context-lang-annotated (lang_6): 6249 examples. We filter LongAlign for samples up to 64k tokens.

Preference Data (Preference)

We used two datasets to train SmolLM3-3B with APO, which has a total of 447k rows. We generated the think equivalent using the prompts of the no_think counterpart and decontaminated using the same methods from the other two stages. The datasets are:

• Tulu 3 8B Preference Mixture (no_think): 231k rows.
• Tulu 3 8B Preference Mixture (think): 216k rows where we generate the chosen responses with Qwen3-32B and the rejected responses with Qwen3-0.6B.

Dataset Stats

The dataset stats contain a more granular level of the training mix by dataset. We also include the Weight column that controls the number of examples we take from each dataset for training. You can find the full configuration files here.

Mid-Training

SFT

Preference Data

License

All the new datasets (aya_dataset-Qwen3-32B, multi-turn-reasoning-if, smolagents-toolcalling-traces, smoltalk-everyday-convs-reasoning-Qwen3-32B, smoltalk-multilingual8-Qwen3-32B, smoltalk-systemchats-Qwen3-32B, table-gpt-Qwen3-32B, tulu_3_8b_pref_mix_qwen3_32b_qwen3_06b_think) are licensed under Apache 2.0. For the existing public datasets, please refer to the original dataset for the license.