AIDev: Studying AI Coding Agents on GitHub (The Rise of AI Teammates in Software Engineering 3.0)

• Papers:
  • The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering
  • AIDev: Studying AI Coding Agents on GitHub
• GitHub: https://github.com/SAILResearch/AI_Teammates_in_SE3

This is AIDev v4 (AIDev-2.7M, cutoff date of Nov 2025). Other versions are available as git tags and can be loaded with load_dataset("hao-li/AIDev", "<config>", revision="<tag>"):

• v3 — previous main: https://huggingface.co/datasets/hao-li/AIDev/tree/v3

📢 Call for Papers

We invite researchers to leverage the AIDev dataset and submit their findings to our upcoming tracks.

Empirical Software Engineering (EMSE) Journal Special Issue (https://emsejournal.github.io/special_issues/2026_SI_Agentic_SE.html)

• Deadline: We are operating on a rolling review process. Submit when ready! (Final date: Sept 30, 2026)

Past Events

KDD 2026 Workshop (https://agent-se.github.io)

• Deadline: June 1, 2026
• Location: Jeju, Korea (August 9 to 13, 2026)

ACM CAIS 2026 Workshop (https://agenticse-cais.github.io)

• Deadline: May 1, 2026 Update: See you in California!
• Location: California, USA (May 26 to 29, 2026)

MSR 2026 (co-located with ICSE) Mining Challenge (https://2026.msrconf.org/track/msr-2026-mining-challenge)

• Deadline: December 23, 2025 Update: See you in Brazil!
• Location: Rio de Janeiro, Brazil (April 12 to 18, 2026)

---

Overview

AIDev is a large-scale dataset capturing the emergence of autonomous coding agents (AI teammates) within real-world open-source software engineering. This version spans 2,743,854 pull requests across 327,477 repositories, authored by six AI coding agents: OpenAI Codex, GitHub Copilot, Cursor, Google Jules, Devin, and Claude Code, and involving 159,056 human developers.

You can easily load the small tables in a few lines of code:

import pandas as pd
all_repo_df = pd.read_parquet("hf://datasets/hao-li/AIDev@v4/all_repository.parquet")
all_user_df = pd.read_parquet("hf://datasets/hao-li/AIDev@v4/all_user.parquet")

⚠️ all_pull_request.parquet is ~1.5 GB on disk and pr_commit_details.parquet is ~1.3 GB — avoid loading them blindly with a plain pd.read_parquet. See Loading the Dataset below for memory-safe patterns.

Intended Uses

• Fine-tuning or post-training: fine-tuning or post-training your LLMs/agents based on the patches
• Empirical SE research: analyse collaboration patterns, review latency, velocity
• Agent evaluation: measure bug-fix success, code quality, PR acceptance rate
• Human–AI interaction: study conversational review dynamics and sentiment

Papers Using AIDev

This year’s MSR Mining Challenge received 130 abstracts (116 full submissions) and accepted 62 papers. The list of accepted papers (all using AIDev) can be found here:

https://2026.msrconf.org/track/msr-2026-mining-challenge

We also list other papers using AIDev below (may overlap with the challenge):

1. Agent READMEs: An Empirical Study of Context Files for Agentic Coding [arXiv:2511.12884]
2. Agentic Refactoring: An Empirical Study of AI Coding Agents [arXiv:2511.04824]
3. Analyzing Message-Code Inconsistency in AI Coding Agent-Authored Pull Requests [arXiv:2601.04886]
4. Do Autonomous Agents Contribute Test Code? A Study of Tests in Agentic Pull Requests [arXiv:2601.03556]
5. Early-Stage Prediction of Review Effort in AI-Generated Pull Requests [arXiv:2601.00753]
6. Security in the Age of AI Teammates: An Empirical Study of Agentic Pull Requests on GitHub [arXiv:2601.00477]
7. How Do Agents Perform Code Optimization? An Empirical Study [arXiv:2512.21757]
8. How Do Agentic AI Systems Address Performance Optimizations? A BERTopic-Based Analysis of Pull Requests [arXiv:2512.24630]
9. A Study of Library Usage in Agent-Authored Pull Requests [arXiv:2512.11589]
10. What Makes a GitHub Issue Ready for Copilot? [arXiv:2512.21426]
11. Understanding Security Risks of AI Agents' Dependency Updates [arXiv:2601.00205]
12. How Do Agentic AI Systems Deal With Software Energy Concerns? A Pull Request-Based Study [arXiv:2512.24636]
13. Will It Survive? Deciphering the Fate of AI-Generated Code inOpen Source [arXiv:2601.16809]
14. Understanding Dominant Themes in Reviewing Agentic AI-authored Code [arxiv:2601.19287]
15. How AI Coding Agents Modify Code: A Large-Scale Study of GitHub Pull Requests [arxiv:2601.17581]

Note: If you use AIDev in your research, please open a Pull Request to add your paper here!

Quick Look

The overview of this version of the AIDev dataset is as follows:

Example Notebooks

Loading the Dataset (without blowing up your RAM)

Three rules keep memory flat:

import pandas as pd
import pyarrow.parquet as pq

# 1. Small tables: load fully, this is fine ("@v4" pins this version; drop it for latest main)
all_repo_df = pd.read_parquet("hf://datasets/hao-li/AIDev@v4/all_repository.parquet")
all_user_df = pd.read_parquet("hf://datasets/hao-li/AIDev@v4/all_user.parquet")

# 2. Big tables: ALWAYS select columns (skip the heavy `title`/`body`/`patch`)
LIGHT_COLS = ["id", "agent", "user", "state", "created_at", "closed_at", "merged_at", "repo_url"]
all_pr_df = pd.read_parquet(
    "hf://datasets/hao-li/AIDev@v4/all_pull_request.parquet", columns=LIGHT_COLS
)

# 3. Or stream record batches for constant-memory processing
pf = pq.ParquetFile("all_pull_request.parquet")  # after hf download
for batch in pf.iter_batches(batch_size=262_144, columns=["agent", "merged_at"]):
    df = batch.to_pandas()
    ...  # aggregate incrementally

See scripts/load_aidev.py for complete memory-safe examples, including streaming patch extraction from pr_commit_details.parquet. The scripts/ folder also contains memory-safe analysis scripts (streaming/column-pruned; they run in well under 1 GB of RAM):

Dataset Structure

AIDev is organized into normalized tables that can be joined via consistent keys. The full-scope tables cover every repository:

• all_pull_request: PR-level data (ID, title, body, agent label, user info, state, timestamps)
• all_repository: Metadata including license, language, stars, forks, and project-level info
• all_user: User information such as id, login, and created date (personally identifiable information has been removed to address privacy concerns)

AIDev-pop: Filtered (>100 stars)

For the AIDev-pop subset (repositories with more than 100 stars) of AIDev, we provide extra tables:

• pull_request: PR-level data (ID, title, body, agent label, user info, state, timestamps)
• repository: Metadata including license, language, stars, forks, and project-level info
• pr_comments & pr_reviews & pr_review_comments: Review discussions, approvals, timestamps, actors; pr_review_comments contains inline review comments
• pr_commits & pr_commit_details: Commit metadata, diffs, file-level changes, patch. Note that the patch data does not include large patches since the GitHub API does not provide them. If you want the large patches, you need to download them yourself.
• user: User information such as id, login, and created date (personally identifiable information has been removed to address privacy concerns)

Dataset Updates

⚠️ Update (v4, cutoff Nov 2025): The dataset has been refreshed to 2.7 million pull requests across six agents (Google Jules added). Earlier versions remain available as git tags (see the version list at the top).

License

This dataset aggregates content from GitHub repositories. Each source repository retains its original copyright and license (e.g., MIT, Apache-2.0, GPL family, Creative Commons variants, etc.). Files, patches/diffs, and any other artifacts originating from those repositories remain governed by their original licenses.

• Users must verify and comply with the specific license of any source repository or file they extract or use from this collection. Do not assume a universal re-license.
• If you believe content appears here in a way that conflicts with its license, please contact the maintainers, and it will be removed.

Important: Repository contents maintain their original licenses. Please respect individual project licenses when using this data.

Citation

If you use AIDev in your work, please cite:

@misc{li2025aiteammates_se3,
  title={The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering}, 
  author={Hao Li and Haoxiang Zhang and Ahmed E. Hassan},
  year={2025},
  eprint={2507.15003},
  archivePrefix={arXiv},
  primaryClass={cs.SE},
  url={https://arxiv.org/abs/2507.15003}
}