Software evolution itineraries (as Milestone DAGs) extracted from real-world repositories for AI agent evaluation. Used by SWE-Milestone. [Paper]

This dataset contains the metadata, task specifications (SRS documents), dependency graphs, and test classifications (e.g., fail_to_pass, pass_to_pass) needed to run SWE-Milestone evaluation trials.

Dataset Statistics

SWE-Milestone covers 7 real-world open-source repositories spanning 5 programming languages, with 98 graded milestones, 110 active dependency edges (84 base + 26 additional), and 48,470 total ΔSrcLoC in graded gold patches.

Column definitions:

• #Milestones - Number of graded milestones: active milestones excluding IDs in non-graded_milestone_ids.txt.
• #Deps - Number of unique edges in the active DAG. It combines dependencies.csv and additional_dependencies.csv, then keeps edges whose two endpoints are active. Non-graded milestones remain part of this DAG because the agent still implements them.
• ΔSrcLoC - Sum of src_loc over graded milestones only. src_loc equals src_additions + src_deletions and excludes test-only changes according to each repository's metadata.
• Src LoC CV - Population coefficient of variation (pstdev / mean) of graded milestone src_loc values.
• Active milestones - IDs from selected_milestone_ids.txt when that file exists; otherwise every ID in milestones.csv.

Dataset Structure

Each repository workspace directory contains:

SWE-Milestone-data/<repo_name>/
├── metadata.json                      # Repo metadata (src_dirs, test_dirs, patterns)
├── dependencies.csv                   # Milestone dependency DAG
├── milestones.csv                     # Milestone catalog
├── selected_milestone_ids.txt         # (optional) Subset of milestones to evaluate
├── additional_dependencies.csv        # (optional) Extra DAG edges
├── non-graded_milestone_ids.txt       # (optional) Milestones excluded from scoring
├── srs/{milestone_id}/SRS.md          # Task specification per milestone
└── test_results/{milestone_id}/       # Test classifications and filters
    ├── {milestone_id}_classification.json
    └── {milestone_id}_filter_list.json   # (optional) Invalid tests excluded from grading

Key Files

• metadata.json --- Repository configuration including source directories, test directory patterns, exclude patterns, and build commands.
• dependencies.csv --- Defines the milestone dependency DAG. Each row is an edge (upstream, downstream, strength).
• milestones.csv --- Catalog of all milestones with IDs, titles, and associated commit ranges.
• srs/{milestone_id}/SRS.md --- Software Requirements Specification describing what the agent needs to implement for each milestone.
• test_results/{milestone_id}/{milestone_id}_classification.json --- Test classifications. The file contains a full classification (all state transitions across 17 categories) and a stable_classification (with flaky tests removed). The evaluator uses stable_classification and only reads fail_to_pass, pass_to_pass, and none_to_pass for grading; the remaining categories (e.g., fail_to_fail, pass_to_skipped, new_tests) are retained for dataset quality analysis but do not affect scoring.
• test_results/{milestone_id}/{milestone_id}_filter_list.json --- (optional) Lists invalid or flaky tests (invalid_fail_to_pass, invalid_none_to_pass) to exclude from grading.

Example: ripgrep Milestone DAG

The figure below shows the milestone DAG for ripgrep (14.1.1 → 15.0.0), illustrating how milestones are structured and connected.

Each milestone in the DAG requires the following components:

• SRS (Software Requirements Specification) --- A Markdown document describing what the agent needs to implement. Located at srs/{milestone_id}/SRS.md.
• Test classification --- A JSON file listing which tests are expected to transition states after the milestone is implemented. Located at test_results/{milestone_id}/{milestone_id}_classification.json. It categorizes tests into:
  • fail_to_pass --- Tests that are currently failing and must pass after the milestone is implemented (the core success criteria).
  • pass_to_pass --- Tests that are currently passing and must remain passing (regression guard).
  • none_to_pass --- New tests introduced by this milestone that should pass. By default, these are merged into fail_to_pass when computing scores (not scored separately).
• Milestone Docker image --- A pre-built Docker image containing the test environment for that specific milestone, used by the evaluator to run tests in isolation. Hosted on DockerHub.
• Base Docker image --- The starting environment where the agent runs, containing the codebase at the start version. Also hosted on DockerHub.

Notes

• The Graded Milestones count includes only milestones that contribute to the final score. Some repositories include additional non-graded milestones (listed in non-graded_milestone_ids.txt) that the agent must still implement as part of the dependency DAG but are excluded from scoring, typically because they are trivial tasks (e.g., version bumps, dependency updates) or lack sufficient test coverage for reliable grading. Only 3 milestones across all repositories are non-graded.
• Each milestone is extracted from the actual commit history of the repository, representing real software evolution between the listed version ranges.

Validation

Run the dataset contract validator from the repository root:

python scripts/validate_data.py

The validator checks the active and graded milestone sets, the effective DAG, SRS and test-classification coverage, classification/filter schemas, and the generated README statistics. It exits non-zero when contract errors are found. After an intentional data change, regenerate the statistics block with:

python scripts/validate_data.py --write-readme

Usage

git lfs install
git clone https://huggingface.co/datasets/DeepCommit-ai/SWE-Milestone-data

Then follow the SWE-Milestone setup guide to run evaluation trials.

Citation

@misc{deng2026swemilestoneevaluatingaiagents,
      title={SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution},
      author={Gangda Deng and Zhaoling Chen and Zhongming Yu and Haoyang Fan and Yuhong Liu and Yuxin Yang and Dhruv Parikh and Rajgopal Kannan and Le Cong and Mengdi Wang and Qian Zhang and Viktor Prasanna and Xiangru Tang and Xingyao Wang},
      year={2026},
      eprint={2603.13428},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2603.13428},
}