Code | Leaderboard | Prior Preference Sets | Results | Paper

Reward Bench Evaluation Dataset Card

The RewardBench evaluation dataset evaluates capabilities of reward models over the following categories:

1. Chat: Includes the easy chat subsets (alpacaeval-easy, alpacaeval-length, alpacaeval-hard, mt-bench-easy, mt-bench-medium)
2. Chat Hard: Includes the hard chat subsets (mt-bench-hard, llmbar-natural, llmbar-adver-neighbor, llmbar-adver-GPTInst, llmbar-adver-GPTOut, llmbar-adver-manual)
3. Safety: Includes the safety subsets (refusals-dangerous, refusals-offensive, xstest-should-refuse, xstest-should-respond, do not answer)
4. Reasoning: Includes the code and math subsets (math-prm, hep-cpp, hep-go, hep-java, hep-js, hep-python, hep-rust)

The RewardBench leaderboard averages over these subsets and a final category from prior preference data test sets including Anthropic Helpful, Anthropic HHH in BIG-Bench, Stanford Human Preferences (SHP), and OpenAI's Learning to Summarize data.

The scoring for RewardBench compares the score of a prompt-chosen pair to a prompt-rejected pair. Success is when the chosen score is higher than rejected.

In order to create a representative, single evaluation score, we perform a limited mixture of averaging across results. For all the subsets detailed below except for Reasoning, we perform per-prompt weighted averaging across all the prompts in the subset to get the section score. For example, in Chat we take a weighted average of the AlpacaEval and MT Bench sets based on the number of prompts. For Reasoning, we increase the weight of the PRM-Math subset so code and math abilities are weighed equally in the final number, rather than increasing the relevance of code. Once all subsets weighted averages are achieved, the final RewardBench score is the average across the subset scores (including Prior Sets).

Dataset Details

In order to maintain all the relevant data, the samples in the dataset will have the following items. Note, the dataset is single-turn:

• prompt (str): the instruction given in the various test sets.
• chosen (str): the response from the better model or the better rated prompt.
• chosen_model (str): where applicable
• rejected (str): the response with the lower score or from word model.
• rejected_model (str): where applicable
• subset (str): the subset (e.g. alpacaeval-easy) of the associated prompt as the dataset is all in one split.
• id (int): an incremented id for every prompt in the benchmark.

To select a specific subset use HuggingFace Datasets .filter functionality.

dataset = dataset.filter(lambda ex: ex["subset"] == "alpacaeval-easy")

This can easily be converted to the standard chosen/rejected list of messages format (see UltraFeedback for an example), for example with our data loading utilities on GitHub.

Subset Summary

Total number of the prompts is: 2985.

The length distribution of the subsets with a Llama tokenizer is shown below.

Filtering Summary

The RewardBench dataset is manually filtered from 5123 source prompts to manually verify the chosen-rejected ranking of prompts.

• The categories of AlpacaEval and MT Bench are manually filtered for every prompt.
• LLMBar, DoNotAnswer, HEP, and Math PRM all contained structured metadata for automatic filtering.
• XSTest is a hybrid of manual confirmation with metadata from the project.
• Refusals are automatically generated as a refusal or response (where refusal is preffered) with manual confirmation.

Substantial filtering details are available in the appendix of the papr. If there are any bugs in the data, please reach out!

License information

Licensing an aggregated dataset is a complex task. We release the RewardBench dataset under ODC-BY requiring the user to follow the licenses of the subsequent parts. Licensing LLM datasets is an evolving topic. The licenses primarily apply to the prompts and the completions generated by models are often unlicensed. The details for the datasets used in this work vary in the level of the detail on licenses and method of applying them.

Within this dataset are prompts created by AI2 (the refusals data, released as MIT for now, see official release soon) with completions from API and open models. More details will come on this soon.

Development

Requirements

Building the dataset requires datasets. Maintaining the script and notebooks requites notebook.

pip install datasets notebook nbconvert

Convert with:

jupyter nbconvert --to script [YOUR_NOTEBOOK].ipynb

With no changes to the ipynb, the dataset can be re-built and pushed with the following (PLEASE BE CAREFUL):

python build_dataset.py

Git LFS notes

If your uploads fail with:

Git LFS upload failed:  14% (1/7), 4.2 MB | 0 B/s                                                                                                                                                 
  (missing) data/train-00000-of-00001.parquet (425c88744455a9b0e7248cdd81fe4716085aae22849798f653f59fc878117a4d)
hint: Your push was rejected due to missing or corrupt local objects.
hint: You can disable this check with: `git config lfs.allowincompletepush true`

First fetch all lfs objects:

git lfs fetch --all origin main

Filtering script (basic)

To filter data, run the following script:

python scripts/filter.py subset-name 0

with a subset from the dataset and a start index.

---

Citation

@misc{RewardBench,
    title={RewardBench: Evaluating Reward Models for Language Modeling},
    author={Lambert, Nathan and Pyatkin, Valentina and Morrison, Jacob and Miranda, LJ and Lin, Bill Yuchen and Chandu, Khyathi and Dziri, Nouha and Kumar, Sachin and Zick, Tom and Choi, Yejin and Smith, Noah A. and Hajishirzi, Hannaneh},
    year={2024},
    howpublished={\url{https://huggingface.co/spaces/allenai/reward-bench}
}