MedSSS Data

💻Github Repo 🖨️arXiv Paper

The official SFT and PRM training data for "MedS3: Towards Medical Slow Thinking with Self-Evolved Soft Dual-sided Process Supervision"

Introduction

The data is a synthetic dataset built from a 8k seed dataset, covering 16 datasets with 5 diverse medical tasks. This dataset is evolved using Monte-Carlo Tree Search, aimed for provided SFT data and PRM data with high quality. This dataset draws from a diverse array of text domains, encompassing knowledge-intensive QA, general biomedical QA, medical NLI, long context QA and reasoning QA. These domains have been carefully selected to represent a wide spectrum of medical knowledge and reasoning, providing a well-rounded foundation for training medical LLMs as well as a fine-grained medical process reward model.

Data Formate

There are two fields, namely 'pos' and 'neg'. Data items in 'pos' field are used for supervised fine-tuning. And each intermediate reasoning step could be utilized for step-wise DPO training and soft dual-sided process reward supervision.

For a given instance:

{
'id': {some_id},
'question': {original question},
'pos': [[[step_1, updated_score, rollout_score], [step_2, updated_score, rollout_score], ..., [step_n, updated_score, rollout_score], final_score, final_score], {another trajectory}],
 'neg': [[[step_1, updated_score, rollout_score], [step_2, updated_score, rollout_score], ..., [step_n, updated_score, rollout_score], final_score, final_score], {another trajectory}],
'answer': {ground truth},
'original_dataset': {dataset_name}
}

Each instance has several trajectories. Each trajectory is a list, consisting of several list items. Each list item contains three elements. The first element is the reasoning step, the second element is the Q-Value after back-propagation, and the third element is the original rollout value. The third value would be sometimes null, as this step is a direct reasoning step following a Finish node. Currently, the datasets library does not support to load this dynamic data format.

For convenient usage, please directly download the sft_2_new.jsonl file to your workspace and process jsonl items directly.