ProgramDistill-Bench

Paper · Blog · Leaderboard

ProgramDistill is a dataset and executable benchmark for evaluating whether coding agents can restore missing behavior in interactive web applications. Each feature-restoration task contains a masked application, a solver-facing task statement, a working reference application configuration, replayable browser traces, a behavioral verifier, and a reference solution.

The release contains 4,063 feature-restoration tasks across 26 applications, a public 300-task model-comparison subset, and 12 full-reconstruction tasks with 590 atomic workflows, 413 cumulative endpoints, and 1,541 cumulative replay stages.

The public release is limited to the packaged benchmark and a dashboard/project page. The ProgramDistill mining, task-generation, and orchestration pipeline is not released. Internal generation records, generation prompts and model configurations, review logs, and agent evaluation trajectories are also outside the release scope. Task-local solver prompts, build recipes, verifiers, and reference solutions remain part of the benchmark tasks described below.

Dataset Details

Dataset Description

ProgramDistill turns working web-application behavior into verifiable coding tasks:

1. Mining: Agents discover application behaviors and retain browser traces that reproduce after reset.
2. Crafting: A task-generation system removes the source implementation for a verified behavior while keeping the application buildable.
3. Patching: A coding agent compares the masked application with a working reference and edits source code to restore the behavior. The original trace is replayed to score the result.

Verified traces can depend on earlier traces, forming prerequisite lineages. This supports atomic feature restoration, cumulative tasks that combine dependent behaviors, and whole-application reconstruction.

• Curated by: Jeonghye Kim and the ProgramDistill research team
• Funded by: Microsoft Research
• Shared by: Microsoft
• Language(s): English (en) for task statements and prompts; source-code languages vary by application
• License: ProgramDistill-authored benchmark artifacts are MIT licensed. Included applications and assets remain subject to their upstream licenses and attribution requirements.
• Dataset snapshot date: 2026-08-25
• Dataset version: 1.0.0

Dataset Sources

• Hugging Face dataset: https://huggingface.co/datasets/microsoft/ProgramDistill
• Dashboard/project page: https://microsoft.github.io/debug-gym/blog/2026/09/programdistill/
• Paper: https://arxiv.org/abs/2609.18805
• Leaderboard: http://microsoft.github.io/debug-gym/blog/2026/09/programdistill/dashboard/

The first release snapshot is identified by:

Uses

Direct Use

ProgramDistill is intended for research on:

• Evaluating coding agents that modify interactive web applications.
• Reference-guided behavioral repair, where a running application forms part of the specification.
• Agents that alternate between browser observation, source-code editing, and runtime validation.
• Atomic and cumulative feature restoration.
• Capability analysis across restoration and prerequisite-lineage depth.
• Whole-application behavioral reconstruction.
• Behavioral verification of the packaged benchmark tasks.

For feature-restoration tasks, success is measured by replaying browser traces and checking their expected signals. Cumulative tasks can award partial credit through chain score, the fraction of dependent behaviors restored. Full-reconstruction tasks are scored against multiple verifier targets.

Out-of-Scope Use

ProgramDistill is not designed or validated for:

• Deploying generated patches or reconstructed applications in production.
• Establishing that a patch is secure, maintainable, accessible, performant, or correct beyond the supplied behavioral verifiers.
• Evaluating non-web software such as mobile, desktop, embedded, command-line, real-time, or safety-critical systems.
• Operating against live third-party services, real user accounts, or systems the user is not authorized to access.
• Vulnerability discovery, exploit development, malware development, or bypassing access controls.
• High-impact decisions about people or organizations.
• Ranking individual developers or drawing conclusions about human developer competence.
• Reproducing or extending the original ProgramDistill mining and task-generation pipeline from released code. The pipeline implementation is not included.

The public partial-reconstruction-subset is not a private or contamination-resistant test set. Every subset task also appears in partial-reconstruction, and reference solutions and verifiers are distributed with the benchmark.

Dataset Structure

Distribution Files

The collection paths below are relative to the repository root.

Collections

Because partial-reconstruction-subset is contained in partial-reconstruction, the two collections must not be added together when counting unique feature-restoration tasks. Collection-level metadata directories are not tasks; each task directory is a standalone Harbor package.

Feature-Restoration Task Layout

A complete example is:

tasks/partial-reconstruction-subset/insurance_claim_web__insurance_claim_web__save_common_accident_claim_draft__logic_and_ui__cumulative/

Each task has this top-level layout:

agent/                 task-local agent and patching prompt
agent_context.json     application, browser, and runtime context
build_images.sh        pinned source materialization and image build
environment/           Harbor environment and application source recipe
gold_gate.sh           reference-solution validation
instruction.md         solver-facing restoration requirement
run.sh                 Harbor agent entry point
solution/              mask diff, masked source, and reference solution
task.toml              task metadata and resource settings
tests/                 masked source, replay traces, verifier, and scorer

The task additionally includes:

• environment/app/images/recipes/<category>/<app>/upstream_pin.json
• environment/app/images/recipes/<category>/<app>/programdistill_overlay.patch
• Masked source under tests/masked/ and solution/masked/
• Mask diff at solution/mask_patch.diff
• Reference solution at solution/solve.sh
• Browser traces and expected signals under tests/traces/
• Verifier and scorer under tests/scorer/

Application source is not represented only as a vendored snapshot. It is reconstructed from the upstream checkout identified by source_repo and source_commit, with programdistill_overlay.patch applied.

Feature-Restoration Data Fields

The principal task.toml fields are:

schema_version
[task] name, description, authors, keywords
[metadata] difficulty, category, task_mode, mask_scope,
           lineage_depth, restoration_depth
[agent] timeout_sec
[environment] network_mode, allowed_hosts, build_timeout_sec,
              cpus, memory_mb, storage_mb
[verifier] timeout_sec, environment_mode
[verifier.environment] docker_image, network_mode, build_timeout_sec,
                       cpus, memory_mb, storage_mb

The principal agent_context.json fields are:

browser_config, browser_port, candidate_id, dataset_type,
dev_start_command, freeze_time, instance_id, ProgramDistill_mode,
readiness_ports, repo_path, repository, submission_excluded_paths

Each upstream_pin.json contains:

app, category, source_repo, source_commit, sparse_keep,
overlay_files, reproduces_vendored, license

The principal browser-trace fields are:

action_trace, artifact_paths, candidate_id, description, errors,
expected_signals, final_observation, freeze_time,
mining_certificate_signature, mining_trace_sha256, name,
parent_trace, quality_labels, retry_history, rows, schema, status,
trace_hint, trace_sequence, validated_at

The broken source state is represented by solution/mask_patch.diff and the files under tests/masked/ and solution/masked/.

Full-Reconstruction Tasks

Each task in tasks/full-reconstruction/ contains:

• A minimal executable application scaffold.
• An application description.
• Frozen source provenance.
• Verifier traces and target configurations.
• Scoring code.
• Runtime configuration.

The 12 tasks contain 590 atomic workflows, 413 cumulative endpoints, and 1,541 cumulative replay stages.

Data Splits

ProgramDistill does not provide conventional train, validation, and test splits:

• partial-reconstruction is the complete 4,063-task feature-restoration corpus.
• partial-reconstruction-subset is a public 300-task evaluation selection from partial-reconstruction.
• full-reconstruction is a separate whole-application reconstruction task type.

The subset includes all 26 applications and uses these restoration-depth quotas:

Researchers creating new splits should group tasks by application and lineage to avoid source and behavioral leakage.

Release Statistics

Counts for the 2026-08-25 task snapshot are:

Verified-behavior discovery records, internal generation records, and saved agent evaluation trajectories are outside the scope of this task release.

Getting Started

Requirements and Setup

Install:

• Git and Git LFS
• Docker Engine or Docker Desktop with Docker Compose
• uv

Start Docker, then install Harbor:

docker info
docker compose version
uv tool install "harbor==0.23.0" --python 3.12
harbor --version

If harbor is not found, run uv tool update-shell and open a new shell.

Download the Executable Benchmark

Enable Git LFS and clone the dataset:

git lfs install
git clone https://huggingface.co/datasets/microsoft/ProgramDistill
cd ProgramDistill

Git LFS downloads large files automatically during cloning. The dataset contains nearly one million files; allow time for checkout and additional disk space for Git/LFS storage and Docker images.

Download Only the 300-Task Subset

Use the Hugging Face CLI to download the subset and its shared runtime manifest:

uv tool install huggingface_hub
hf download microsoft/ProgramDistill \
  --repo-type dataset \
  --include "tasks/partial-reconstruction-subset/**" \
  --include "README.md" \
  --include "pyproject.toml" \
  --include "tasks/README.md" \
  --local-dir ProgramDistill-subset
cd ProgramDistill-subset
find tasks/partial-reconstruction-subset -type f -name "*.sh" -exec chmod +x {} +

hf download does not preserve executable permissions or symlinks. The last command restores shell-script permissions; use Git if you need the original file modes and symlinks.

Run and Score a Task

From the cloned repository or downloaded subset directory, validate the reference solution:

TASK=tasks/partial-reconstruction-subset/insurance_claim_web__insurance_claim_web__save_common_accident_claim_draft__logic_and_ui__cumulative

"$TASK/gold_gate.sh"

This checks the provided solution without a model or API key.

Then run the bundled solver:

export ANTHROPIC_API_KEY="<your-api-key>"
"$TASK/run.sh" --model anthropic/claude-sonnet-4-5

Both wrappers build images automatically; run.sh also installs the shared runtime. Model names use LiteLLM's provider/model format.

Optional ten-step smoke test:

MINEPATCH_MAX_STEPS=10 "$TASK/run.sh" --model anthropic/claude-sonnet-4-5

The sample defaults to 1,000 steps and a 10-hour agent timeout. A short smoke test may score 0.0 without making edits; it is not a full solving attempt.

For Full Reconstruction:

TASK=tasks/full-reconstruction/baserow_baserow
"$TASK/gold_gate.sh"
"$TASK/run.sh" --model anthropic/claude-opus-5

Inspect Results and Diagnose Failures

Results default to jobs/<job-name>/. Both wrappers accept --jobs-dir and --job-name.

Check the recorded errors and reward, not just the exit code: Harbor can exit with status 0 after a setup failure. The reference solution should score 1.0. For model runs, exit_reason: abs_step_limit means the step cap was reached, not necessarily that the task was solved.

Dataset Creation

Curation Rationale

Most software-engineering benchmarks derive tasks from written issues and hidden tests. This is expensive to scale, emphasizes codebases with suitable issue histories, and does not directly test whether an agent can infer intended behavior from running software.

ProgramDistill instead treats working application behavior as an executable specification. It creates automatically verified tasks that exercise both browser interaction and source-code modification, with difficulty scaling through behavioral dependencies.

Source Data

ProgramDistill began with 30 candidate web applications from OSWorld application sources and public open-source repositories. The final feature-restoration release includes 26 applications.

The following candidates are not included:

• dinogame_web
• glbviewer_web
• slidepuzzle_web
• calcom_cal_com

For every released application, each task carries frozen provenance at:

environment/app/images/recipes/<category>/<app>/upstream_pin.json
environment/app/images/recipes/<category>/<app>/programdistill_overlay.patch

upstream_pin.json records the ProgramDistill application ID, source category, upstream repository URL, source commit, sparse checkout paths, overlay files, vendoring-reproduction status, and license. The overlay records ProgramDistill modifications.

Initial application state is reconstructed from the pinned upstream source together with the ProgramDistill overlay, fixtures, and reset scripts included with the task.

Data Collection and Processing

Each application is packaged as a stable reference instance and a hot-reloading editable instance. Application state is reset before collection and replay, and a deterministic clock reduces runtime variation.

The following summarizes how the released tasks were created; the construction pipeline implementation is not included in the public release:

1. Proposes grounded feature goals from application and source evidence.
2. Executes goals in a live browser and records the actions actually taken.
3. Relabels each trace according to the behavior actually achieved.
4. Re-collects and replays candidate traces after reset.
5. Masks the source implementation for each verified behavior.
6. Retains a task only when the original behavior passes, the masked application builds and runs, and the behavior fails after masking.
7. Rejects superficial masks that leave the substantive implementation intact.
8. Packages the task, pinned provenance, source overlay, verifier, and reference solution in Harbor-compatible form.

The release contains the resulting packaged benchmark tasks, not the ProgramDistill mining, task-generation, or orchestration pipeline. Internal generation model and prompt versions, manual review records, generation timelines, verified-behavior discovery records, and evaluation trajectories are not part of the release. Task-local prompts under each task's agent/ directory are included because they are required to execute that packaged task.

Who are the source data producers?

The original application source was produced by OSWorld application authors and contributors to the included public open-source repositories. ProgramDistill task statements, masks, traces, verifier metadata, and reference solutions were generated by the ProgramDistill pipeline.

The release is not based on human study participants, crowdworkers, or human annotation of personal behavior.

Annotations

Annotation Process

ProgramDistill annotations are generated automatically. They include behavior names and descriptions, action traces, expected signals, prerequisite relationships, mask scopes, task metadata, and verification outcomes.

Validation is executable:

• Success signals come from observed application state.
• Behavior identities reflect achieved outcomes rather than intended goals.
• Clean traces must replay after reset.
• Masked applications must build and run while causing their original trace to fail.
• gold_gate.sh validates the packaged reference solution.

Replay establishes that a trace is reproducible against the packaged application. It does not prove that every label is the only possible semantic description of the behavior.

Who are the annotators?

Annotations are machine-generated by the ProgramDistill agent pipeline and deterministic browser, replay, masking, and scoring infrastructure. The current task release does not include internal generation or review records.

Personal and Sensitive Information

ProgramDistill does not collect data from live users or study participants. It processes selected public open-source repositories and task-local fixtures in self-contained benchmark applications.

Public repositories can contain contributor names, public attribution, copyright notices, and contact information required by upstream licenses. Application fixtures can include person-like names, messages, accounts, or credentials used only inside the benchmark. Seeded credentials must not grant access to external systems.

The release should not contain private repositories, live credentials, secrets, or private user data. Reports of unexpected personal or sensitive content should be sent to the maintained dataset contact for review and removal.

Bias, Risks, and Limitations

• Application coverage: The release contains 26 web applications and cannot represent the full diversity of software, frameworks, accessibility needs, deployment environments, or user workflows.
• Language: Task statements and most application interfaces are in English.
• Web-only scope: ProgramDistill does not cover mobile, desktop, embedded, command-line, real-time, or safety-critical software.
• Public evaluation data: partial-reconstruction-subset is fully public, contained in partial-reconstruction, and distributed with solutions and verifiers. It is not a hidden test set and may be present in model training data after release.
• No conventional splits: The release does not define train, validation, or held-out test partitions.
• Exploration bias: The tasks reflect behaviors found by the mining system, not an exhaustive specification of each application.
• Observable-verifier scope: Passing a verifier establishes only that its actions and expected signals were recovered. It does not establish complete functional equivalence, source equivalence, security, accessibility, performance, maintainability, or correctness in unobserved states.
• Generated tasks: Task descriptions and masks are machine-generated. Executable gates reduce unsupported or shallow tasks but cannot guarantee ideal labels or masking.
• Lineage dependence: Cumulative tasks are correlated. Failure to restore an early prerequisite can block later behaviors.
• Deterministic runtime: Frozen state and a deterministic clock improve replay but do not model production concurrency, live data, or changing external services.
• Bounded observation settling: Network and DOM quiescence are bounded heuristics rather than proof that every asynchronous effect has completed.
• Visual coverage: The pipeline emphasizes structured browser observations and replay-stable selectors; visual-only behavior can be underrepresented.
• Upstream dependency: Source materialization depends on pinned upstream repositories remaining available and compatible with the task build recipe.
• Upstream licensing: ProgramDistill-authored benchmark artifacts are MIT licensed. Included applications and assets retain their upstream licenses and attribution requirements.
• Pipeline availability: The ProgramDistill mining, task-generation, and orchestration pipeline is not released. Users can materialize, run, verify, and inspect the packaged benchmark tasks, but cannot regenerate the corpus from scratch using the original pipeline.
• Generation provenance scope: Internal model versions, generation prompts and configurations, review decisions, discovery records, and evaluation trajectories are not part of this release. Task-local solver prompts are included with the benchmark.
• Executable-code risk: Repositories, build scripts, reference solutions, and model-generated patches are executable, untrusted code.

Recommendations

• Report the dataset version or task snapshot commit with all results.
• Use tasks/partial-reconstruction-subset/ for direct comparison with reported 300-task evaluations and tasks/partial-reconstruction/ for broader coverage.
• Do not describe partial-reconstruction-subset as held out from partial-reconstruction.
• If creating new partitions, split by application and keep related masks and prerequisite lineages together.
• Run tasks and generated patches only in isolated, resource-limited containers or virtual machines.
• Follow each task's network policy and use only local benchmark credentials.
• Ensure the agent cannot read solution/, hidden evaluator assets, or other excluded paths during an evaluation run.
• Treat source, assets, build scripts, downloads, model outputs, and patches as untrusted content.
• Run gold_gate.sh after materializing a task to validate the packaged reference solution.
• Preserve and follow each upstream application's license and attribution requirements.
• Do not infer production readiness, security, maintainability, or broad software-engineering competence from ProgramDistill scores alone.

Citation

BibTeX:

@article{programdistill2026,
  author        = {Jeonghye Kim and Minseon Kim and Young Jin Kim and
                   Matheus Pereira and Marc-Alexandre C{\^o}t{\'e} and
                   Alessandro Sordoni and Xingdi Yuan and Zhengyan Shi},
  title         = {ProgramDistill: From Interactive Web Apps to Verifiable
                   Reference-Guided SWE Tasks},
  year          = {2026},
  eprint        = {2609.18805},
  archivePrefix = {arXiv},
  primaryClass  = {cs.SE},
  url           = {https://arxiv.org/abs/2609.18805}
}

Glossary

• Atomic task: Feature-restoration task targeting one verified behavior.
• Cumulative task: Feature-restoration task combining dependent behaviors along a prerequisite lineage.
• Behavior trace: Replayable browser actions plus expected signals.
• Reference application: Working feature-intact application used to infer intended behavior.
• Current application: Writable masked application edited by the agent.
• Chain score: Fraction of behaviors in a cumulative target chain restored.
• Logic-only mask: Removes behavior while preserving the feature interface.
• Logic-and-UI mask: Removes behavior and feature-specific interface markup.
• Full Reconstruction: Whole-application reconstruction task scored against multiple verifier targets.

Dataset Card Authors

Jeonghye Kim and the ProgramDistill research team.

Dataset Card Contact

• Maintained project contact: jeonghye.kim@kaist.ac.kr, froggy@microsoft.com
• Technical-report contact: Jeonghye Kim (jeonghye.kim@kaist.ac.kr)