CuratedThoughts: Data Curation for RL Training Datasets

Andreas Hochlehnert¹*, Hardik Bhatnagar¹*, Vishaal Udandarao¹, Ameya Prabhu¹, Matthias Bethge¹

* - shared first authors, ¹ - Tübingen AI Center, University of Tübingen

Since the release of DeepSeek-R1 and its reinforcement learning-based reasoning approach, new datasets such as Open-R1 and OpenThoughts have been introduced. These datasets are used for various post-training approaches like supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO).

What is GRPO

Group Relative Policy Optimization (GRPO), as implemented in DeepSeek-R1, involves sampling from a group of model outputs and using verifiable rewards to score them. Outputs with higher rewards are then preferred, and the model's policy is updated accordingly. Through this method, the model is guided into generating reasoning traces that assist in solving a given task.

What Are the Issues?

To generate proper reasoning traces, outputs must be both verifiable and unequivocal. Through a series of simple tests, we have demonstrated that these datasets might not be ideal for reinforcement learning-based training. Specifically, we identified the following issues:

1. Questions Contain Subquestions. Instead of providing a single verifiable answer, some questions include subquestions, where only one is answered in the answer column. This could lead to incorrect reasoning traces, as answers to the other subquestions are marked as wrong.
2. Questions Seeking a Mathematical Proof. Requesting a proof is problematic due to the lack of automated methods for verifying correctness. The answer column typically contains only the initial equation to be proved. For GRPO training, there is no effective mechanism to enforce correct proof behavior, which may result in incorrect reasoning traces as long as the model returns whatever proof was presented in the question.
3. Questions Referring to Figures. Some questions refer to figures that are not included in the dataset, making them difficult to handle for text-only models.
4. Questions that contain None as the ground truth (GT). Some questions have their ground truth solution set as None.

Here is an overview of the percentage for Open-R1 and OpenThoughts:

Examples

Subquestions

Three, (10 points) Let $a+b+c=0, a^{2}+b^{2}+c^{2}=1$.
(1) Find the value of $ab+bc+ca$;
(2) Find the value of $a^{4}+b^{4}+c^{4}$....

Example 6 The rules of a "level-up game" stipulate: On the $n$-th level, a die must be rolled $n$ times. If the sum of the points obtained from these $n$ rolls is greater than $2^{n}$, the level is considered passed. Questions: (1) What is the maximum number of levels a person can pass in this game? (2) What is the probability that he can pass the first three levels consecutively?

Proof

Prove that number $1$ can be represented as a sum of a finite number $n$ of real numbers, ...

Figure

Augusto has a wire that is $10 \mathrm{~m}$ long. He makes a cut at a point on the wire, obtaining two pieces. One piece has a length of $x$ and the other has a length of $10-x$ as shown in the figure below:

How Did We Test?

These are the patterns we used for filtering:

# Match actual subquestions (mid-text questions)
subq_pattern = r"""
	(?<=\n)                             # Ensure it appears after a newline
	(?:                                 # Start non-capturing group for subquestion markers
		\(\s*[1-9a-zA-Zivx]+\s*\)|      # (1), (a), (i), (ii), etc.
		(?:Part|Question)\s+[a-zA-Z0-9] # "Part A", "Question 1", etc.
	)
	\s*                                 # Optional whitespace
	[A-Za-z0-9]                         # Ensure it's followed by actual content
"""

proof_pattern = r"\b(?:prove|proof|show that|demonstrate that|verify that)\b(?!.*multiple choice)"


figure_pattern = r"""
	\b(?:
		(?:as\s+)?shown\s+in\s+(?:the\s+)?(?:figure|diagram|picture)|
		see\s+(?:the\s+)?(?:figure|diagram|picture)|
		(?:figure|diagram|picture)\s+(?:below|above|\d+)|
		in\s+(?:the\s+)?(?:figure|diagram|picture)
	)\b
"""

We also ensured that it is not a multiple-choice question, to reduce false positives

mcq_pattern = r"""
	(?:^|\n)        # Start of line or newline
	\(\s*[A-E]\s*\) # (A), (B), etc.
	.*?             # Some text
	(?:\n|$)        # End of line or end of string
"""

blank_mcq_pattern = r"""
	\(\s*\)\s*   # Blank parentheses for missing MCQ answers
	.*?          # Some text
	\([A-E]\)    # Actual MCQ options
"""

How Can We Improve?

We present here a filtered version of open-r1/OpenR1-Math-220k, open-thoughts/OpenThoughts-114k and open-r1/OpenThoughts-114k-math that enhances their suitability for GRPO-based training.

Cite

@misc{curatedthoughts,
  author = {Hochlehnert, Andreas and Bhatnagar, Hardik and Udandarao, Vishaal and Prabhu, Ameya and Bethge, Matthias},
  title  = {CuratedThoughts: Data Curation for RL Training Datasets},
  url    = {https://huggingface.co/datasets/bethgelab/CuratedThoughts},
  year   = {2025},
}