Dataset Descriptions

This dataset contains 19890 preference examples and is synthetically created to enhance models' precise instruction following capabilities while satisfying several constraints. The dataset containts preference pairs (chosen, reject responses) and can be used for preference tuning methods (e.g., PPO, DPO).

Dataset Construction

To create this dataset, we took a subset of its supervised-tuning version here and convert it into preference dataset. Specifically, we rewrote each prompt in the subset to relax one of the given constraints such that the response to the modified prompt is no longer a valid response for the original prompt. We use the response to the new modified prompt as the rejected response.

• Paper: TBD
• Repository: TBD
• Language(s) (NLP): English
• License: ODC-BY
• Point of Contact: Faeze Brahman

Loading

from datasets import load_dataset

dataset = load_dataset("allenai/tulu-3-pref-personas-instruction-following")["train"]

Dataset Structure

Each example in the dataset contains the standard instruction-tuning data points as follow:

• id (str): a unique identifier
• prompt (str): python programming questions grounded in a given persona/scenario
• constraints (list of str): a list of verifiable constraints that need to be satisfied by the assistant response
• chosen (str): the chosen response for the given instruction following prompt satisfying the constraints
• rejected (str): the rejected response failing to satisfy one of the given constraints
• chonsen_model (str): model used to generate the chosen response
• rejected_model (str): model used to generate the rejected response