cisc839-assignment-1 -- review effort analysis of agentic pull requests

table of contents

• overview
• key features
• setup
• usage
• deliverables
• references
• contributing
• author

overview

code reviewers spend significant effort on agentic (ai-authored) pull requests, but it is unclear whether that effort differs from human-authored prs. this project analyzes 27,640 merged prs from the aidev-pop dataset (msr'26 mining challenge) to answer: do agentic prs require more review effort than human prs?

the pipeline runs five staged analyses -- rq refinement, eda, hypothesis testing, ols regression, and a genai-augmented reanalysis -- using standard statistical methods (mann-whitney u, cliff's delta, statsmodels ols) on real github pr data.

the analysis finds agentic prs merge significantly faster overall (p < 0.001, cliff's delta = -0.581), with copilot as the sole exception, requiring more review effort than human prs by all three measures. the genai-augmented pipeline (part 3) adds task-type controls, reducing omitted variable bias and increasing model r-squared from 0.142 to 0.156.

key features

• end-to-end pipeline: five notebooks cover rq refinement, eda, hypothesis testing, regression, and summary -- each notebook is independently runnable after setup.
• non-parametric testing with effect size: mann-whitney u + cliff's delta reported for overall and per-agent comparisons, with bonferroni correction for multiple tests.
• ols regression with model comparison: two baseline models (binary + per-agent) and three genai-augmented models, with model design rationale and diagnostic notes (jarque-bera).
• honest data limitations: human pr review data is structurally absent in aidev; the report documents the fillna(0) directional bias and proposes the correct fix.
• reproducible replication package: all notebooks, source code, preprocessed data, and ai prompt log packaged in a single zip.

setup

0. prerequisites

• git
• anaconda or miniconda
• python 3.12

1. clone the repository

git clone https://github.com/elsayedelmandoh/cisc839-assignment-1
cd cisc839-assignment-1

2. create conda environment

conda create -n dataqueens python=3.12 -y
conda activate dataqueens
conda install pip -y
pip install -r requirements.txt

3. environment variables

create a .env file at the project root (see .env.example for required keys):

HF_TOKEN=your_huggingface_token_here

a huggingface token is required to download the aidev-pop dataset in 00-quickstart.ipynb.

usage

run the notebooks in order:

jupyter notebook

deliverables

docs/02-deliverables/
  A1_Report_elmandoua_elsayed.pdf          -- 4-page report (font 11)
  A1_Replication_Elmandoua_elsayed.zip     -- replication package (notebooks + excel log)
  AI_Prompt_Log_Elmandoua_elsayed.xlsx     -- ai prompt log (purpose, prompt, how used)

references

full references are in docs/00-assignment1/01-references.md.

contributing

contributions are welcome. to contribute:

1. fork the repository.
2. create a branch (git checkout -b feature/my-change).
3. commit with clear messages (git commit -m 'add some feature').
4. push to your fork (git push origin feature/my-change).
5. open a pull request.

author

elsayed elmandoh - nlp engineer

• connect on linkedin and x via linktree