Dataset Summary

DebugBench is a Large Language Model (LLM) debugging benchmark introduced in the paper DebugBench: Evaluating Debugging Capability of Large Language Models. We collect code snippets from the LeetCode community and implant bugs into source data with GPT-4. The project is also open-sourced as a GitHub repository.

• It consists of 4,253 instances.
• It covers four major bug categories and 18 minor types.
• It includes C++, Java, and Python instances.
• It contains three difficulty levels: easy, medium, and hard.
• All the instances were released after June 2022.
• Please refer to the article for more details.

Data Fields

An instance in DebugBench contains 13 features.

Data Splits

The dataset is an evaluation benchmark and there comprises only one split, the eval split of 4, 253.

Evaluation

The evaluation is based on the unseen test suites from LeetCode, a popular programming challenge platform. The evaluator will need a leetcode account for the usage of test suites. To obtain the 'leetcode_session' cookie, they may utilize the developer view in their web browsers like Chrome or use browser extensions like EditThisCookie.

Please refer to the following code example for evaluation. This example is accessible in this Hugging Face repository. The leetcode_env package comes from Leetcode Hard Gym. More implementation details are available in our GitHub repository.

import os
from .leetcode_env.environment import LeetCodeEnv
from .leetcode_env.types import LeetCodeSubmission, ProgrammingLanguage

LEETCODE_SESSION_COOKIE = os.environ['LEETCODE_SESSION']
class LeetCodeTester(object):

    def __init__(self):
        os.environ['LEETCODE_SESSION'] = LEETCODE_SESSION_COOKIE
        self.env = LeetCodeEnv(cooldown=15)
        self.lang_dict = {
            "python3": ProgrammingLanguage.PYTHON3,
            "java": ProgrammingLanguage.JAVA,
            "cpp": ProgrammingLanguage.CPP,
        }

    def test(self, code: str, task_id: str, language: str) -> tuple[bool, dict]:
        lang = self.lang_dict.get(language)
        sub = LeetCodeSubmission(code=code, lang=lang, question_slug=task_id)
        status, reward, done, submission_result = self.env.step(sub)
        return reward, submission_result

if __name__ == '__main__':

    tester = LeetCodeTester()
    task_id = "make-number-of-distinct-characters-equal"
    code = "class Solution:\n\n    def insertAndRemove(self, mp, toInsert..."  # abbreviated
    print(tester.test(code, task_id, "python3"))

Here are two output examples.

(True, {'status_code': 10, 'lang': 'python3', 'run_success': True, 'status_runtime': '111 ms', 'memory': 18488000, 'question_id': '2615', 'elapsed_time': 133, 'compare_result': '111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111', 'code_output': '', 'std_output': '', 'last_testcase': '', 'expected_output': '', 'task_finish_time': 1704769340887, 'task_name': 'judger.judgetask.Judge', 'finished': True, 'total_correct': 99, 'total_testcases': 99, 'runtime_percentile': 73.75, 'status_memory': '18.5 MB', 'memory_percentile': 15.625, 'pretty_lang': 'Python3', 'submission_id': '1141026534', 'status_msg': 'Accepted', 'state': 'SUCCESS'})
(False, {'status_code': 11, 'lang': 'python3', 'run_success': True, 'status_runtime': 'N/A', 'memory': 18532000, 'question_id': '2615', 'elapsed_time': 184, 'compare_result': '101110111101010010111100110101111111011010100001111101011111000111010111000111101011011011101110011', 'code_output': 'false', 'std_output': '', 'last_testcase': '"abcc"\n"aab"', 'expected_output': 'true', 'task_finish_time': 1704769355341, 'task_name': 'judger.judgetask.Judge', 'finished': True, 'total_correct': 64, 'total_testcases': 99, 'runtime_percentile': None, 'status_memory': 'N/A', 'memory_percentile': None, 'pretty_lang': 'Python3', 'submission_id': '1141026664', 'input_formatted': '"abcc", "aab"', 'input': '"abcc"\n"aab"', 'status_msg': 'Wrong Answer', 'state': 'SUCCESS'})

Dataset Creation

As illustrated in the figure above, to construct DebugBench, we collect code snippets from the LeetCode community, implant bugs into source data with GPT-4, and assure quality checks. We also evaluate two commercial and three open-source models in a zero-shot scenario. Please refer to the article for more details.

Limitation

• Bug instances in our experiments are synthetically created and might not entirely reflect the intricacies of real-world debugging scenarios.
• For a few bug subtypes, some bug instances may have an inconsistent taxonomy nature than the labeled feature. An example of this is a bug generated in response to a prompt specifying 'undefined methods.' Rather than invoking undefined functions as expected, the code triggers an infinite loop.

Citation Information

@misc{tian2024debugbench,
      title={DebugBench: Evaluating Debugging Capability of Large Language Models}, 
      author={Runchu Tian and Yining Ye and Yujia Qin and Xin Cong and Yankai Lin and Zhiyuan Liu and Maosong Sun},
      year={2024},
      eprint={2401.04621},
      archivePrefix={arXiv},
      primaryClass={cs.SE}
}