ELI5 paired This is a processed version of the eli5 dataset. The dataset was created following very closely the steps in the stack-exchange-paired dataset. The following steps were applied:

• Create pairs (response_j, response_k) where j was rated better than k
• Sample at most 10 pairs per question
• Shuffle the dataset globally

This dataset is designed to be used for preference learning using techniques such as Reinforcement Learning from Human Feedback. The processing notebook is in the repository as well.

If you want to construct a "question" column in this data, you can either use just the "title" column, or concatenate the "title" column with the "selftext" column as follows:

def get_question(example):
  title = example["title"]
  selftext = example["selftext"]
  if selftext:
      if selftext[-1] not in [".", "?", "!"]:
          seperator = ". "
      else:
          seperator = " "
      question = title + seperator + selftext
  else:
      question = title
  example["question"] = question
  return example

dataset = load_dataset("vincentmin/eli5_askscience_askhistorians_rlhf")
dataset = dataset.map(get_question)

For the license, see the eli5 dataset which states "The licensing status of the dataset hinges on the legal status of the Pushshift.io data which is unclear." at the time of creation of this dataset.