DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving

📝 Paper@arXiv | 🤗 Datasets&Models@HF | 🐱 Code@GitHub

🐦 Thread@X(Twitter) | 🐶 中文博客@知乎 | 📊 Leaderboard@PapersWithCode | 📑 BibTeX

Models: DART-Math

DART-Math models achieve performance superior or competitive to previous SOTAs on 2 in-domain and 4 challenging out-of-domain mathematical reasoning benchmarks, despite using much smaller datasets and no proprietary model like GPT-4.

Abbreviations: College (CollegeMath), DM (DeepMind Mathematics), Olympiad (OlympiadBench-Math), Theorem (TheoremQA). Bold means the best score by SFT on the respective base model here. To reproduce our results, please refer to the DART-Math GitHub repository.

Prompt Template

All the DART-Math models use the Alpaca prompt template:

Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n###Instruction:\n{query}\n\n### Response:\n

Training Dataset

We construct our traning datasets by applying Difficulty-Aware Rejection Sampling (DARS) to the MATH and GSM8K training sets.

DARS tackle severe biases towards easy queries, with frequent failures to generate any correct response for the most challenging queries, in previous datasets.

These biases are primarily caused by vanilla rejection sampling, where the same number of responses is sampled for each query, yet the likelihood of obtaining correct responses for difficult queries is significantly lower, sometimes even zero.

Please refer to DART-Math-Hard / DART-Math-Uniform for more details.

Training Setup

We perform standard instruction tuning to several base models including Llama3-8B & Mistral-7B & Llama3-70B as representatives of general models and DeepSeekMath- 7B as the representative of math-specialized model on our synthetic datasets DART-Math-Hard & DART-Math-Uniform, leading to DART-Math (Prop2Diff) & DART-Math (Uniform) respectively.

For simplicity, we keep most hyper-parameters the same across different models and datasets:

• Model max length (of packed sequence): 4096
• Batch size: 64
• Warm-up ratio: 0.03
• Learning rate scheduler: cosine
• Prompt template: Alpaca

Several other key hyper-parameters are tuned as follow:

• For maximum learning rate, we determine the values by searching through 1e-6,5e-6,1e-5,2e-5,5e-5,1e-4 according to the MATH performance after training on MMIQC for 1 epoch, except for Llama3-70B that is so expensive to search for that we derive from Llama3-8B’s learning rate in analogy to the relationship of (per-training) learning rates between Llama2-7B and Llama2-70B (~2:1).
• For Llama3 models, preliminary experiments indicate that training for 1 epoch consistently outperforms 3 epochs.

Please refer to Appendix A.1 of our paper for more details.

Other Details

• For Mistral-7B-based models, we disable sliding_window by default following the newest Mistral-7B-Instruct (Flash Attention 2 does not support sliding_window and XFormer backend in vLLM has throughput ~10% lower in our experiments.)

Citation

If you find our data, model or code useful for your work, please kindly cite our paper:

@article{tong2024dartmath,
  title={DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving},
  author={Yuxuan Tong and Xiwen Zhang and Rui Wang and Ruidong Wu and Junxian He},
  year={2024},
  eprint={2407.13690},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2407.13690},
}