BigCodeBench

Dataset Description

• Homepage: https://bigcode-bench.github.io/
• Repository: https://github.com/bigcode-project/bigcodebench
• Paper: Link
• Point of Contact: contact@bigcode-project.org terry.zhuo@monash.edu

The dataset has 2 variants:

1. BigCodeBench-Complete: Code Completion based on the structured docstrings.
2.  BigCodeBench-Instruct: Code Generation based on the NL-oriented instructions.

The overall statistics of the dataset are as follows:

The function-calling (tool use) statistics of the dataset are as follows:

Changelog

Dataset Summary

BigCodeBench is an easy-to-use benchmark which evaluates LLMs with practical and challenging programming tasks. The dataset was created as part of the BigCode Project, an open scientific collaboration working on the responsible development of Large Language Models for Code (Code LLMs). BigCodeBench serves as a fundamental benchmark for LLMs instead of LLM Agents, i.e., code-generating AI systems that enable the synthesis of programs from natural language descriptions as well as others from code snippets.

Languages

The dataset only contains natural language in English and programming language in Python (3.0+).

How to use it

from datasets import load_dataset

# full dataset
ds = load_dataset("bigcode/bigcodebench", split="v0.1.4")

# dataset streaming (will only download the data as needed)
ds = load_dataset("bigcode/bigcodebench", streaming=True, split="v0.1.4")
for sample in iter(ds): 
    print(sample)

Dataset Structure

Data Fields

• task_id (string): The unique identifier for the task.
• complete_prompt (string): The PEP257-structured docstring prompt.
• instruct_prompt (string): The natural-language-oriented instruction prompt.
• canonical_solution (string): The canonical solution w/o comments.
• code_prompt (string): The code-only prompt.
• test (string): The code snippet for testing, wrapped in a unittest.TestCase class.
• entry_point (string): The entry point for the code snippet, which is task_func.
• doc_struct (string[dictionary]): The structured docstring.
  • description (string): The main task description in natural language.
  • note (string): The additional notes for the task in natural language.
  • reqs (string, optional): The modules can be used in the task solution.
  • params (string, optional): The parameters used in the task solution.
  • returns (string, optional): The values to be returned in the task solution.
  • raises (string, optional): The exceptions should be raised in the task solution.
  • examples (string, optional): The interactive Python examples as hints for the task solution.
• libs (string): The libraries can be used in the task solution.

Data Splits

The dataset has no splits, and all data is loaded as train split by default.

Dataset Creation

For more information on the dataset construction, please refer to the technical report. GitHub Action pipeline code is available here.

Curation Rationale

We believe that there are three main expectations of a good execution-based programming benchmark:

1. The benchmark should be easy to use and efficient in evaluating the fundamental capabilities of LLMs. Repo-level and agent-centric benchmarks (e.g., SWE-bench) are not suitable for this purpose.
2. The benchmark should be practical, covering various programming scenarios. Algo-specific benchmarks (e.g., HumanEval and MBPP) are unsuitable. Domain-specific benchmarks (e.g., DS-1000) are also unsuitable for this purpose.
3. The benchmark should be challenging, where the tasks require LLMs' strong compositional reasoning capabilities and instruction-following capabilities. The benchmarks with simple tasks (e.g., ODEX) are unsuitable.

BigCodeBench is the first benchmark that meets all three expectations. It is an easy-to-use benchmark that evaluates LLMs with challenging and practical programming tasks, accompanied by an end-to-end evaluation framework bigcodebench. We aim to assess how well LLMs can solve practical and challenging programming tasks in an open-ended setting.

Source Data

Data Collection

For the dataset construction, please refer to Section 2 in technical report.

Who are the source language producers?

The data was originally sourced from GPT-4-0613, with the seed examples from ODEX (collected from StackOverflow). The data was then annotated through the collaboration between human experts and LLMs.

Considerations for Using the Data

Discussion of Biases

We agree that there could be a few programming tasks with slightly biased instructions or over-specific test cases. Considering that software development is iterative, incremental, and collaborative, we believe that the bias can be mitigated with the long-term development of BigCodeBench and additional help from the open-source community. We are open to feedback and suggestions for improving the dataset.

Other Known Limitations

See Appendix D in technical report for more information. We highlight a few limitations as follows:

• Multilingualism
• Saturation
• Reliability
• Efficiency
• Rigorousness
• Generalization
• Evolution
• Interaction

Additional Information

Dataset Curators

1. Terry Yue Zhuo, Monash University & CSIRO's Data61, terry.zhuo@monash.edu
2. Other BigCode Project members include
   • Minh Chien Vu
   • Jenny Chim
   • Han Hu
   • Wenhao Yu
   • Ratnadira Widyasari
   • Imam Nur Bani Yusuf
   • Haolan Zhan
   • Junda He
   • Indraneil Paul
   • Simon Brunner
   • Chen Gong
   • Thong Hoang
   • Armel Randy Zebaze
   • Xiaoheng Hong
   • Wen-Ding Li
   • Jean Kaddour
   • Ming Xu
   • Zhihan Zhang
   • Prateek Yadav
   • Niklas Muennighoff

Licensing Information

BigCodeBench is licensed under the Apache License, Version 2.0.

Citation Information

@article{zhuo2024bigcodebench,
  title={BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions},
  author={Zhuo, Terry Yue and Vu, Minh Chien and Chim, Jenny and Hu, Han and Yu, Wenhao and Widyasari, Ratnadira and Yusuf, Imam Nur Bani and Zhan, Haolan and He, Junda and Paul, Indraneil and others},
  journal={arXiv preprint arXiv:2406.15877},
  year={2024}
}