FT-LLM 2026 Reasoning DPO

A Direct Preference Optimization (DPO) dataset of (prompt, chosen, rejected) triples used in Phase 2 of the COMPASS project to align a Japanese VLM's LLM backbone toward correct mathematical reasoning. The chosen responses are chain-of-thought traces distilled from a Qwen3-30B teacher in the structured <Problem>/<Thinking>/<Answer> XML format. The rejected responses are synthetically generated by corrupting the chosen responses under three strategies, mixed at roughly equal weight:

• omit_thinking — remove the contents of the <Thinking> tag
• tamper_thinking_numbers — corrupt numerical values inside the reasoning
• tamper_answer — change the final answer while leaving the reasoning intact

Part of the Compass collection.

License

Released under the Apache License 2.0.

Note on source materials and Japanese copyright law: Under Article 30-4 of the Japanese Copyright Act, the use of copyrighted works for the purpose of information analysis — including machine learning training — is a permitted use that does not require authorization from, or trigger license conditions of, the copyright holders. This dataset was produced in Japan on that basis, and the resulting artifacts are redistributed under Apache-2.0. Downstream users are responsible for complying with any applicable terms of the original source datasets and the teacher model's usage policy in their own jurisdiction.