All rights and obligations of the dataset are with original authors of the paper/dataset. I have merely made this dataset with a MIT licence available on HuggingFace.

BIG-Bench Hard Dataset

This repository contains a copy of the BIG-Bench Hard dataset. Small edits to the formatting of the dataset are made to integrate it into the Inspect Evals repository, a community contributed LLM evaulations for Inspect AI a framework by the UK AI Safety Institute.

The BIG-Bench Hard dataset is a collection of various task categories, with each task focused on testing specific reasoning, logic, or language abilities. The dataset also includes two types of 3-shot prompts for each task: answer-only prompts and chain-of-thought prompts.

Dataset Structure

Main Task Datasets

The collection includes a wide range of tasks, with each designed to evaluate different aspects of logical reasoning, understanding, and problem-solving abilities. Below is a list of all included tasks:

1. Boolean Expressions

   • Evaluate the truth value of a Boolean expression using Boolean constants (True, False) and basic operators (and, or, not).

2. Causal Judgment

   • Given a short story, determine the likely answer to a causal question about the story based on moral, intentional, or counterfactual analysis.

3. Date Understanding

   • Manipulate and reason about dates in various formats by converting date formats, calculating intervals, and answering related questions.

4. Disambiguation QA

   • Resolve ambiguous pronouns or determine if a pronoun’s reference is inherently ambiguous, identifying the correct antecedent where possible.

5. Dyck Languages

   • Predict the sequence of closing parentheses for a Dyck-4 word sequence, given an incomplete set of parentheses.

6. Formal Fallacies Syllogisms Negation

   • Assess logical validity in informal arguments, with a focus on understanding deductive validity versus formal fallacies involving negations.

7. Geometric Shapes

   • Given an SVG path with multiple commands, determine the resulting geometric shape.

8. Hyperbaton (Adjective Ordering)

   • Determine the grammatically correct sentence from two English sentences with different adjective orders.

9. Logical Deduction

   • Deduce the order of a sequence of objects based on clues about spatial relationships and placements.

10. Movie Recommendation

    • Recommend a new movie based on a user's viewing history from four potential choices.

11. Multi-Step Arithmetic

    • Solve multi-step arithmetic equations involving basic operations like addition, subtraction, multiplication, and division.

12. Navigate

    • Predict whether an agent will return to its starting point after a series of navigation steps.

13. Object Counting

    • Given a list of possessions with quantities, determine the total count of a specific object class (e.g., fruits).

14. Penguins in a Table

    • Answer attribute-related questions about penguins based on a unique table format, sometimes with additional context.

15. Reasoning about Colored Objects

    • Answer questions about the color of objects based on contextual information.

16. Ruin Names

    • Identify a humorous one-character edit to the name of an artist, band, or movie.

17. Salient Translation Error Detection

    • Determine the type of error in the English translation of a German source sentence.

18. Snarks

    • Distinguish between two nearly-identical sentences to identify which one is sarcastic.

19. Sports Understanding

    • Judge whether a factitious sentence about sports is plausible.

20. Temporal Sequences

    • Based on a series of daily activities, determine when the person might have been free for another activity.

21. Tracking Shuffled Objects

    • Track the final positions of objects after a series of pairwise swaps from an initial arrangement.

22. Web of Lies

    • Evaluate the truth value of a Boolean function expressed as a natural-language word problem.

23. Word Sorting

    • Sort a list of words lexicographically.

Each dataset contains:

• question: The task question text
• choices: Multiple choice options
  • label: List of choice identifiers (A, B, C, etc.)
  • text: List of choice texts
• target: Correct answer label

Few-Shot Prompts

The few_shot_prompts dataset provides example prompts for each task type with two formats:

• answer_only_prompt: Direct answer template
• chain_of_thought_prompt: Template encouraging step-by-step reasoning

Usage

Loading the Dataset

from datasets import load_dataset

# Load a specific task
date_dataset = load_dataset("Joschka/big_bench_hard", "date_understanding")

# Load prompts
prompts = load_dataset("Joschka/big_bench_hard", "few_shot_prompts")

Using Few-Shot Prompts

def get_task_prompts(prompts_dataset, task_name):
    prompt_data = prompts_dataset['few_shot_prompts'].filter(
        lambda x: x['dataset_name'] == task_name
    )[0]
    return {
        'answer_only': prompt_data['answer_only_prompt'],
        'chain_of_thought': prompt_data['chain_of_thought_prompt']
    }

# Get prompts for date understanding task
date_prompts = get_task_prompts(prompts, 'date_understanding')

Data Files

Each dataset configuration includes its own data files:

• boolean_expressions/boolean_expressions-*
• causal_judgment/causal_judgment-*
• date_understanding/date_understanding-*
• disambiguation_qa/disambiguation_qa-*
• dyck_languages/dyck_languages-*
• formal_fallacies_syllogisms_negation/formal_fallacies_syllogisms_negation-*
• geometric_shapes/geometric_shapes-*
• hyperbaton/hyperbaton-*
• logical_deduction/logical_deduction-*
• movie_recommendation/movie_recommendation-*
• multi_step_arithmetic/multi_step_arithmetic-*
• navigate/navigate-*
• object_counting/object_counting-*
• penguins_in_a_table/penguins_in_a_table-*
• reasoning_about_colored_objects/reasoning_about_colored_objects-*
• ruin_names/ruin_names-*
• salient_translation_error_detection/salient_translation_error_detection-*
• snarks/snarks-*
• sports_understanding/sports_understanding-*
• temporal_sequences/temporal_sequences-*
• tracking_shuffled_objects_five_objects/tracking_shuffled_objects_five_objects-*
• tracking_shuffled_objects_seven_objects
• tracking_shuffled_objects_three_objects/tracking_shuffled_objects_three_objects-*
• web_of_lies/web_of_lies-*
• word_sorting/word_sorting-*

Citation

If your research makes use of this dataset please cite the BIG-Bench Hard paper.

BIG-Bench Hard (Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them (Suzgun et al., 2022))

@article{suzgun2022challenging,
  title={Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them},
  author={Suzgun, Mirac and Scales, Nathan and Sch{\"a}rli, Nathanael and Gehrmann, Sebastian and Tay, Yi and Chung, Hyung Won and Chowdhery, Aakanksha and Le, Quoc V and Chi, Ed H and Zhou, Denny and and Wei, Jason},
  journal={arXiv preprint arXiv:2210.09261},
  year={2022}
}

BIG-Bench Hard, focuss on a suite of 23 challenging BIG-Bench tasks which we call BIG-Bench Hard (BBH). These are the task for which prior language model evaluations did not outperform the average human-rater. We find that applying chain-of-thought (CoT) prompting to BBH tasks enables PaLM to surpass the average humanrater performance on 10 of the 23 tasks, and Codex (code-davinci-002) to surpass the average human-rater performance on 17 of the 23 tasks. Since many tasks in BBH require multi-step reasoning, few-shot prompting without CoT, as done in the BIG-Bench evaluations (Srivastava et al., 2022), substantially underestimates the best performance and capabilities of language models, which is better captured via CoT prompting. As further analysis, we explore the interaction between CoT and model scale on BBH, finding that CoT enables emergent task performance on several BBH tasks with otherwise flat scaling curves.

BIG Bench (Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models (Srivastava et al., 2022))

@article{srivastava2022beyond,
  title={Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models},
  author={Srivastava, Aarohi and Rastogi, Abhinav and Rao, Abhishek and Shoeb, Abu Awal Md and Abid, Abubakar and Fisch, Adam and Brown, Adam R and Santoro, Adam and Gupta, Aditya and Garriga-Alonso, Adri{\`a} and others},
  journal={arXiv preprint arXiv:2206.04615},
  year={2022}
}

Related Research

This dataset is referenced in the survey paper A Survey of Deep Learning for Geometry Problem Solving, where it is discussed in the context of geometry problem solving tasks, specifically its geometric_shapes subtask. The accompanying GitHub repository for this survey, which maintains a list of relevant papers and datasets, can be found here: https://github.com/majianz/gps-survey.

Abstract

Geometry problem solving is a key area of mathematical reasoning, which is widely involved in many important fields such as education, mathematical ability assessment of artificial intelligence, and multimodal ability assessment. In recent years, the rapid development of deep learning technology, especially the rise of multimodal large language models, has triggered a widespread research boom. This paper provides a survey of the applications of deep learning in geometry problem solving, including (i) a comprehensive summary of the relevant tasks in geometry problem solving; (ii) a thorough review of related deep learning methods; (iii) a detailed analysis of evaluation metrics and methods; and (iv) a critical discussion of the current challenges and future directions that can be explored. Our goal is to provide a comprehensive and practical reference of deep learning for geometry problem solving to promote further developments in this field. We create a continuously updated list of papers on GitHub: this https URL .

Abstract

BIG-Bench (Srivastava et al., 2022) is a diverse evaluation suite that focuses on tasks believed to be beyond the capabilities of current language models. Language models have already made good progress on this benchmark, with the best model in the BIG-Bench paper outperforming average reported human-rater results on 65% of the BIG-Bench tasks via few-shot prompting. But on what tasks do language models fall short of average human-rater performance, and are those tasks actually unsolvable by current language models?

License

This dataset is licensed under MIT.