CRAVE: Code Review Agent Verdict Evaluation

Dataset Description

The CRAVE dataset is a balanced, filtered code review classification dataset containing 1,200 samples from 123 repositories and 600 pull requests. This dataset has been filtered and selected for high quality, making it specifically designed for training and evaluating code review agents that can classify pull request changes as either APPROVE or REQUEST_CHANGES.

Dataset Summary

• Total Samples: 1,200
• Unique Repositories: 123
• Unique Pull Requests: 600
• Label Distribution: {'APPROVE': 600, 'REQUEST_CHANGES': 600}

Dataset Structure

Features

• original_pull_request_url: Direct URL to the GitHub pull request
• pull_request_title: Title of the pull request
• repo: GitHub repository (owner/name format)
• pr_number: Pull request number
• diff: JSON object containing base and head commit hashes for generating the patch
• patch: The actual code diff content for review
• label: Ground truth verdict (APPROVE or REQUEST_CHANGES)
• explanation: Explanation of the verdict decision
• description: Original pull request description (from pull_requests.body)
• hint: Task description generated by LLM (from tasks.task)

Data Splits

The dataset is provided as a single unified file (dataset.parquet) to allow maximum flexibility for users. Researchers can create their own splits based on their specific needs:

• Repository-based splits: Split by repository to avoid data leakage
• Temporal splits: Split by creation date for time-based evaluation
• Random splits: Traditional random train/validation/test splits
• Stratified splits: Ensure balanced label distribution across splits

Optional pre-defined splits are also provided as separate files:

• train.parquet: ~960 samples (80%)
• validation.parquet: ~120 samples (10%)
• test.parquet: ~120 samples (10%)

These splits are based on repository hash to ensure consistent, deterministic splits.

Dataset Sources

Data Collection

The dataset was created by:

1. Collecting pull requests from diverse open-source repositories
2. Extracting code diffs and metadata
3. A set of heuristic rules are used to select meaningful code reviews with back-and-forth discussion between human author and reviewers
4. Validating labels through automated and manual review processes

Repository Distribution

Top repositories by sample count:

• apache/kafka: 74 samples
• yt-dlp/yt-dlp: 58 samples
• keras-team/keras: 38 samples
• sveltejs/svelte: 36 samples
• tokio-rs/tokio: 36 samples
• LAION-AI/Open-Assistant: 34 samples
• microsoft/TypeScript: 34 samples
• ray-project/ray: 34 samples
• crewAIInc/crewAI: 32 samples
• hyprwm/Hyprland: 32 samples

Change Type Distribution

• fix: 528 samples
• feature: 456 samples
• refactor: 94 samples
• doc: 70 samples
• test: 20 samples
• performance: 12 samples
• config: 12 samples
• style: 6 samples
• security: 2 samples

Uses

Intended Use Cases

• Code Review Classification: Train models to classify pull request changes
• Automated Code Review: Develop AI agents for code review assistance
• Research: Study code review patterns and human-AI collaboration
• Benchmarking: Evaluate code review models and techniques

Limitations

• Domain Bias: Focus on open-source projects, may not represent enterprise codebases
• Human Labeling Bias: Human reviewers may have bias based on hidden context of the repositories and tasks

@dataset{CRAVE,
  title={CRAVE: Code Review Agent Verdict Evaluation},
  author={Li Zhang},
  year={2025},
  url={https://huggingface.co/datasets/TuringEnterprises/CRAVE},
  note={Code review classification dataset from curated human code reviews}
}

Usage

Loading the Dataset

from datasets import load_dataset
from sklearn.model_selection import train_test_split

# Option 1: Load unified dataset and create custom splits
dataset = load_dataset("TuringEnterprises/CRAVE")
full_data = dataset["train"]  # All data is in the "train" split

# Create custom splits (e.g., by repository)
repos = list(set(full_data["repo"]))
train_repos = repos[:int(len(repos) * 0.8)]
val_repos = repos[int(len(repos) * 0.8):int(len(repos) * 0.9)]
test_repos = repos[int(len(repos) * 0.9):]

train_data = full_data.filter(lambda x: x["repo"] in train_repos)
val_data = full_data.filter(lambda x: x["repo"] in val_repos)
test_data = full_data.filter(lambda x: x["repo"] in test_repos)

# Option 2: Use pre-defined splits (if available)
# train_dataset = load_dataset("TuringEnterprises/CRAVE", split="train")
# val_dataset = load_dataset("TuringEnterprises/CRAVE", split="validation") 
# test_dataset = load_dataset("TuringEnterprises/CRAVE", split="test")

Example Usage

# Basic usage
from datasets import load_dataset

dataset = load_dataset("TuringEnterprises/CRAVE", split="train")

# Access a sample
sample = dataset[0]
print(f"Repository: {sample['repo']}")
print(f"PR URL: {sample['original_pull_request_url']}")
print(f"Title: {sample['pull_request_title']}")
print(f"Label: {sample['label']}")
print(f"Hint: {sample['hint']}")
print(f"Description: {sample['description'][:200]}...")

Dataset Card Authors

• Li Zhang: Dataset creation and curation
• Contact: lilin.wang@turing.com, ashni.sheth@turing.com

Dataset Card Contact

For questions about this dataset, please contact lilin.wang@turing.com or open an issue in the dataset repository.

---

This dataset card was generated on 2025-10-07 00:44:57