Simple Self-Distillation

✨ Overview

This repository reproduces the method from the paper:

Embarrassingly Simple Self-Distillation Improves Code Generation

The approach consists of three simple steps:

1. Sample solutions from a frozen model at non-unit temperature
2. Fine-tune on raw, unverified outputs using standard cross-entropy
3. Decode with a separately tuned temperature

No rewards · No verifier · No teacher · No RL

For full details, see the paper.

📰 News

• [2026-04-03] 🚀 Initial release of repository
• [2026-04-03] 🤗 Model checkpoints coming soon on Hugging Face
• [2026-04-07] 🤗 Model checkpoints released
• [2026-04-16] 🔧 Data generation pipeline released
• (More updates will be added here)

🚀 Getting Started

git clone https://github.com/apple/ml-ssd.git
cd ml-ssd
uv sync --group evaluation          # for evaluation only
uv sync --group data-generation     # for data generation only
uv sync --group evaluation --group data-generation  # for both

source .venv/bin/activate
python evaluation/eval.py \
    --model <hf_model_name> \
    --tensor_parallel_size 4 \
    --max_tokens 65536 \
    --n_repeat 10 \
    --sampling_params "temperature=0.9,top_p=0.8,top_k=20" \
    --output_path ./results/

source .venv/bin/activate
python data_generation/generate.py --config data_generation/config.yaml

🤗 Models

📁 Repository Structure

├── data_generation/
│   ├── generate.py              # End-to-end data generation pipeline
│   ├── config.yaml              # Generation & post-processing config
│   └── templates/               # Prompt templates
├── evaluation/
│   ├── eval.py                  # CLI entry point
│   ├── benchmark.py             # LiveCodeBench v6 implementation
│   └── livecodebench_utils.py   # Code execution utilities
├── figures/
│   └── fig_teaser.png
├── pyproject.toml
└── README.md

📝 Citation

@misc{zhang2026embarrassinglysimpleselfdistillationimproves,
      title={Embarrassingly Simple Self-Distillation Improves Code Generation},
      author={Ruixiang Zhang and Richard He Bai and Huangjie Zheng and Navdeep Jaitly and Ronan Collobert and Yizhe Zhang},
      year={2026},
      eprint={2604.01193},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2604.01193},
}