🦁 WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild

Loading

from datasets import load_dataset
wb_data = load_dataset("allenai/WildBench", "v2", split="test")

Quick Links:

• HF Leaderboard
• HF Dataset
• Github

Dataset Description

• License: CC BY

• Language(s) (NLP): English

• Point of Contact: Yuchen Lin

WildBench is a subset of WildChat. The use of WildChat data to cause harm is strictly prohibited.

Data Fields

The dataset on Hugging Face is organized with several features, each of which is designed to capture specific information pertinent to the data being represented. Here is a descriptive breakdown of each feature:

• id: A unique identifier for each entry, represented as an integer (int64). Not often used.

• session_id: A string that uniquely identifies an example, which is usually used as id.

• conversation_input: A list structure that encompasses multiple attributes related to the input of the conversation:

  • content: The actual text content of the conversation input, stored as a string.
  • language: A string indicating the language used in the conversation input.
  • redacted: A boolean flag (bool) to denote whether any part of the content has been redacted for privacy or other reasons.
  • role: A string indicating the role of the party in the conversation (e.g., 'user', 'assistant').
  • toxic: A boolean indicating whether the content contains any toxic elements.

• references: A list of dict items.

  • gpt-4: The value is the gpt-4 generation as the assistant to the next turn.

• checklist: A sequence of strings that could represent a set of questions to evaluate the outputs.

• length: An integer (int64) representing the length of the conversation or content. Note that this is the number of messages.

• primary_tag: A string that labels the entry with a primary category.

• secondary_tags: A sequence of strings providing additional categorizations.

• intent: A string indicating the underlying intent of the conversation or the interaction instance.

• appropriate: A string that assesses or describes whether the conversation or content is considered appropriate, potentially in terms of content, context, or some other criteria.

Introduction of the WildBench Leaderboard

Licensing Information

WildBench is made available under the CC BY license. It is intended for research and educational use in accordance with Ai2's Responsible Use Guidelines.

Citation

@article{yuchen2024wildbench,
  title={WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild},
  author={Yuchen Lin, Bill and Deng, Yuntian and Chandu, Khyathi and Brahman, Faeze and Ravichander, Abhilasha and Pyatkin, Valentina and Dziri, Nouha and Le Bras, Ronan and Choi, Yejin},
  journal={arXiv e-prints},
  pages={arXiv--2406},
  year={2024}
}

https://arxiv.org/abs/2406.04770