ContextBench

This repository provides:

• default: the full ContextBench table (single train split).
• contextbench_verified: a 500-instance subset (single split).

Columns

The dataset uses a unified schema across sources:

• instance_id: ContextBench instance id (e.g., SWE-Bench-Verified__python__...).
• original_inst_id: Original benchmark instance id (e.g., astropy__astropy-14539).
• source: One of Verified, Pro, Poly, Multi.
• language: Programming language.
• repo_url: Repository URL (from curated annotations).
• base_commit: Base commit sha.
• gold_context: JSON-encoded list of span objects. Each element has file, start_line, end_line, content.
• patch, test_patch: Reference patches.
• problem_statement, f2p, p2p: Source benchmark fields where available.

How gold_context is built

Gold context is constructed from curated annot.json files:

• Sorted by file then by (start_line, end_line) within each file.
• Stored as a JSON list of span objects:
  • file: file path
  • start_line, end_line: line range
  • content: extracted text

Usage

from datasets import load_dataset

ds_full = load_dataset("Schwerli/ContextBench", "default")
ds_subset = load_dataset("Schwerli/ContextBench", "contextbench_verified")