CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding

CodeJudgeBench is a benchmark aimed at evaluating LLM-based judges for coding related tasks.

Updates

• [13/02/2026] 🚀 CodeJudgeBench Adversarial is out — see LINK.

Quickstart

from datasets import load_dataset

prompt = """
Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user prompt displayed below. You will be given assistant A's answer and assistant B's answer. Your job is to evaluate which assistant's answer is better.

After providing your explanation, you must output only one of the following choices as your final verdict with a label:

1. Assistant A is better: [[A>B]] 
2. Assistant B is better: [[B>A]]

Example output: "My final verdict is Assistant A is better: [[A>B]]".

<|User Prompt|>
{query}

<|The Start of Assistant A's Answer|>
{answer_a}
<|The End of Assistant A's Answer|>

<|The Start of Assistant B's Answer|>
{answer_b}
<|The End of Assistant B's Answer|>
""".strip()

task = 'codegen'
split = 'gemini_2.5_pro'
dataset = load_dataset("mattymchen/codejudgebench", task)[split]
sample = dataset[0]
print(prompt.format(
    query=sample['question_content'],
    answer_a=sample['pos_response'],
    answer_b=sample['neg_response']
))

Data Instance and Fields

CodeJudgeBench comprises three subsets: codegen, coderepair, and testgen. Each subset contains multiple splits, featuring responses generated by various LLMs, e.g., claude_3.7.

codegen

• question_content: The coding question
• pos_response: The correct response
• neg_response: The incorrect response

codegen_pass5

• question_content: The coding question
• pos_response: The list of correct response(s)
• neg_response: The list of incorrect response(s)

coderepair

• question_content: The coding question
• wrong_code: The code to be repaired
• wrong_meta: The error message
• pos_response: The correct response
• neg_response: The incorrect response

testgen

• question_content: The coding question
• pos_input: The input for the correct unit test
• neg_input: The input for the incorrect unit test
• pos_response: The correct test output
• neg_response: The incorrect test output

License

The dataset is available under the Apache 2.0 license.

Citation

If you find CodeJudgeBench useful or relevant to your work, please kindly cite our paper:

@article{jiang2025codejudgebench,
  title   = {CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks},
  author  = {Hongchao Jiang and Yiming Chen and Yushi Cao and Hung-yi Lee and Robby T. Tan},
  year    = {2025},
  journal = {arXiv preprint arXiv: 2507.10535}
}