Dataset Card for Paper Review Dataset (ICLR 2023-2025)

Dataset Description

This dataset contains paper submissions and review data from the International Conference on Learning Representations (ICLR) for the years 2023, 2024, and 2025. The data is sourced from OpenReview, an open peer review platform that hosts the review process for top ML conferences.

Focus on Review Data

This dataset emphasizes the peer review ecosystem surrounding academic papers. Each record includes comprehensive review-related information:

• Related Notes (related_notes): Contains review discussions, meta-reviews, author responses, and community feedback from the OpenReview platform
• Full Paper Content: Complete paper text in Markdown format, enabling analysis of the relationship between paper content and review outcomes
• Review Metadata: Structured metadata including page statistics, table of contents, and document structure analysis

The review data captures the full peer review workflow:

• Initial submission reviews from multiple reviewers
• Author rebuttal and response rounds
• Meta-reviews from area chairs
• Final decision notifications (Accept/Reject)
• Post-publication discussions and community comments

This makes the dataset particularly valuable for:

• Review Quality Analysis: Studying patterns in peer review quality and consistency
• Decision Prediction: Building models to predict acceptance decisions based on paper content and reviews
• Review Generation: Training models to generate constructive paper reviews
• Bias Detection: Analyzing potential biases in the peer review process
• Scientific Discourse Analysis: Understanding how scientific consensus forms through discussion

Dataset Structure

Data Instances

Each instance represents a paper with its associated review data:

{
  "id": "RUzSobdYy0V",
  "title": "Quantifying and Mitigating the Impact of Label Errors on Model Disparity Metrics",
  "authors": "Julius Adebayo, Melissa Hall, Bowen Yu, Bobbie Chern",
  "abstract": "Errors in labels obtained via human annotation adversely affect...",
  "year": "2023",
  "conference": "ICLR",
  "related_notes": "[Review discussions, meta-reviews, and author responses]",
  "pdf_url": "https://openreview.net/pdf?id=RUzSobdYy0V",
  "source_url": "https://openreview.net/forum?id=RUzSobdYy0V",
  "content": "[Full paper text in Markdown format]",
  "content_meta": "[JSON metadata with TOC and page statistics]"
}

Data Fields

Review Data Structure

The related_notes field contains the complete review history from OpenReview, stored as a Python tuple string (not JSON format). It includes:

1. Paper Decision: Final acceptance/rejection decision with detailed rationale
2. Primary Reviews: Detailed reviews from multiple reviewers
3. Meta-Reviews: Summary and recommendations from area chairs
4. Author Responses: Rebuttals and clarifications from authors
5. Discussion Threads: Back-and-forth discussions between reviewers and authors
6. Community Comments: Post-decision feedback and questions

Important Notes:

• The related_notes field is stored as a Python tuple string (uses single quotes, Python literals like None)
• Parse with eval(related_notes) to get a tuple of dictionaries
• 100% parseable using eval()

Format Variations:

• 2023 papers: Direct values (e.g., {'decision': 'Accept: poster'})
• 2024/2025 papers: Values wrapped in dicts (e.g., {'decision': {'value': 'Accept (poster)'}})

Dataset Creation

Curation Rationale

This dataset was created to enable research on understanding and improving the peer review process in machine learning conferences. By combining full paper content with complete review discussions, researchers can:

• Analyze how paper characteristics relate to review outcomes
• Study the language and patterns in constructive reviews
• Build systems to assist reviewers or authors
• Investigate fairness and bias in peer review

Source Data

The data was collected from the OpenReview platform, which hosts the ICLR review process in an open format. All reviews, discussions, and decisions are publicly available on the OpenReview website.

Data Processing

1. Paper Content Extraction: Full papers were converted to Markdown format from PDF sources
2. Review Aggregation: Review discussions were extracted from OpenReview forums
3. Quality Filtering: Records with missing essential fields (ID, content, or related_notes) were removed
4. Metadata Extraction: Structural metadata (TOC, page statistics) was extracted from papers

Considerations for Using the Data

Social Impact of the Dataset

This dataset provides transparency into the peer review process, which is typically opaque. By making reviews and discussions publicly available, it enables:

• Analysis of review quality and consistency
• Identification of potential biases in evaluation
• Development of tools to assist the review process
• Educational resources for understanding peer review

However, users should be aware that:

• Reviews represent subjective opinions of reviewers
• Reviewer identities are not included to protect privacy
• Reviews should be interpreted within the context of the specific conference and time period

Discussion of Biases

The dataset may contain several biases:

• Reviewer Bias: Different reviewers may have different standards and tendencies
• Conference-Specific Norms: ICLR review norms may differ from other venues
• Temporal Shifts: Review criteria may have evolved across 2023-2025
• Selection Bias: Papers in this dataset represent ICLR submissions, which may not generalize to all ML research

Other Known Limitations

• Reviewer identities are anonymized to protect privacy
• The related_notes field is stored as Python tuple string (not JSON) and requires eval() for parsing
• Format varies between years (2023 vs 2024/2025) - see usage examples for handling both formats

Usage Examples

Loading the Dataset

import json

# Load from JSONL
with open('ICLR_merged_cleaned_huggingface.jsonl', 'r', encoding='utf-8') as f:
    for line in f:
        paper = json.loads(line)

        print(f"Title: {paper['title']}")
        print(f"Year: {paper['year']}")
        print(f"Review Data: {paper['related_notes'][:200]}...")
        break

Analyzing Review Content

# Parse related_notes to extract decision and reviews
def extract_decision(related_notes):
    """Extract paper decision handling both 2023 and 2024/2025 formats"""
    reviews = eval(related_notes)  # Parse Python tuple string

    for entry in reviews:
        content = entry.get('content', {})
        if not isinstance(content, dict):
            continue

        # Handle both format variations
        if 'decision' in content:
            decision = content['decision']
            if isinstance(decision, str):
                return decision  # 2023 format
            elif isinstance(decision, dict) and 'value' in decision:
                return decision['value']  # 2024/2025 format

    return None

def extract_all_reviews(related_notes):
    """Extract all review entries"""
    reviews = eval(related_notes)
    return list(reviews)  # Convert tuple to list

# Example usage
import json

with open('ICLR_merged_cleaned_huggingface.jsonl', 'r', encoding='utf-8') as f:
    for line in f:
        paper = json.loads(line)

        # Extract decision
        decision = extract_decision(paper['related_notes'])
        print(f"Title: {paper['title']}")
        print(f"Decision: {decision}")

        # Extract all reviews
        all_reviews = extract_all_reviews(paper['related_notes'])
        print(f"Number of review entries: {len(all_reviews)}")

        # Access individual reviews
        for i, review in enumerate(all_reviews):
            content = review.get('content', {})
            title = content.get('title', 'N/A')
            print(f"  Review {i+1}: {title}")

        break

Additional Information

Dataset Curators

This dataset was compiled by AgentAlphaAGI from publicly available OpenReview data.

Licensing Information

The papers and reviews in this dataset are subject to the copyright and terms of use of the OpenReview platform and the respective authors.

Citation Information

If you use this dataset, please cite:

@article{xu2026idea2story,
  title={Idea2Story: An Automated Pipeline for Transforming Research Concepts into Complete Scientific Narratives},
  author={Xu, Tengyue and Qian, Zhuoyang and Liu, Gaoge and Ling, Li and Zhang, Zhentao and Wu, Biao and Zhang, Shuo and Lu, Ke and Shi, Wei and Wang, Ziqi and others},
  journal={arXiv preprint arXiv:2601.20833},
  year={2026}
}

Acknowledgments

This dataset would not be possible without the open peer review platform provided by OpenReview and the contributions of the ICLR community.