OpenJudge Benchmark Dataset

Benchmark dataset for evaluating graders across text, multimodal, and agent scenarios. This dataset supports the OpenJudge framework with labeled preference pairs for quality-assured grader development.

Dataset Statistics

Evaluation Benchmarks

Training Data

Dataset Structure

# Evaluation Benchmarks
text/{task_type}/{task_type}_eval_v1.json
multimodal/{task_type}/{task_type}_eval_v1.json  
agent/{task_category}/{task_name}.json

# Training Data
train_rm/bradley_terry/{train,test}.parquet
train_rm/sft/{train,test}.parquet
train_rm/grpo/pointwise/{train,val}.parquet
train_rm/grpo/pairwise/{train,val}.parquet

Data Format

Each JSON file contains an array of evaluation cases:

{
  "id": "unique_identifier",
  "dataset": "source_dataset_name",
  "task_type": "evaluation_task_type",
  "input": {
    "query": "user query or null",
    "context": "additional context or structured data",
    "reference": "ground truth or reference response",
    "media_contents": [],
    "metadata": {}
  },
  "chosen": {
    "response": {
      "content": "preferred response",
      "model": "model_name",
      "model_type": "text|multimodal",
      "metadata": {}
    }
  },
  "rejected": {
    "response": {
      "content": "dis-preferred response",
      "model": "model_name",
      "model_type": "text|multimodal",
      "metadata": {}
    }
  },
  "human_ranking": [0, 1],
  "metadata": {
    "source": "source_information"
  }
}

Key Fields:

• input: Query, context, reference answer
• chosen/rejected: Preference pair responses (may be null for agent data)
• human_ranking: Preference ranking [chosen_idx, rejected_idx]
• metadata: Task-specific metadata

Notes:

• Text/Multimodal: Standard preference pairs with chosen and rejected
• Agent: Context contains trajectory data; either chosen or rejected may be null

Training Data Format

Bradley-Terry (train_rm/bradley_terry/*.parquet):

SFT (train_rm/sft/*.parquet):

GRPO Pointwise (train_rm/grpo/pointwise/*.parquet):

GRPO Pairwise (train_rm/grpo/pairwise/*.parquet):

Usage

from datasets import load_dataset

# Load entire dataset
dataset = load_dataset("agentscope-ai/OpenJudge")

# Load evaluation benchmarks
text_data = load_dataset("agentscope-ai/OpenJudge", data_files="text/**/*.json")
agent_data = load_dataset("agentscope-ai/OpenJudge", data_files="agent/**/*.json")
multimodal_data = load_dataset("agentscope-ai/OpenJudge", data_files="multimodal/**/*.json")

# Load training data
bt_train = load_dataset("agentscope-ai/OpenJudge", data_files="train_rm/bradley_terry/train.parquet")
sft_train = load_dataset("agentscope-ai/OpenJudge", data_files="train_rm/sft/train.parquet")

# Load GRPO training data
grpo_pointwise = load_dataset("agentscope-ai/OpenJudge", data_files="train_rm/grpo/pointwise/train.parquet")
grpo_pairwise = load_dataset("agentscope-ai/OpenJudge", data_files="train_rm/grpo/pairwise/train.parquet")

Task Categories

Text: Correctness, Hallucination, Harmlessness, Instruction Following, Relevance

Multimodal: Image Coherence, Image Editing, Image Helpfulness, Text-to-Image

Agent: Action Alignment, Memory (Accuracy/Retrieval/Preservation), Plan Feasibility, Reflection (Accuracy/Awareness/Understanding), Tool Use (Selection/Parameters/Success)

Reproduce Evaluation Results

Each task directory contains an evaluate_*.py script that allows you to reproduce the accuracy results using the corresponding OpenJudge grader.

Run Single Evaluation

# Set environment variables
export OPENAI_API_KEY=your_api_key
export OPENAI_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1

# Run specific grader evaluation
cd text/correctness
python evaluate_correctness.py --model qwen-max

# Run with verbose output
python evaluate_correctness.py --model qwen-max --verbose

Run All Evaluations (Parallel)

Use run_all_evaluations.py to evaluate all graders concurrently:

# Run all evaluations
python run_all_evaluations.py

# Run specific category
python run_all_evaluations.py --category text
python run_all_evaluations.py --category agent

# Custom models and concurrency
python run_all_evaluations.py --text-model qwen-max --agent-model qwen3-max --workers 5

# Save results to JSON
python run_all_evaluations.py --output results.json

Expected Accuracy by Grader

License

Apache 2.0

Citation

@software{openjudge2025,
  title  = {OpenJudge: A Unified Framework for Holistic Evaluation and Quality Rewards},
  author = {The OpenJudge Team},
  url    = {https://github.com/modelscope/OpenJudge},
  year   = {2025}
}

Links

• GitHub: modelscope/OpenJudge
• Documentation: modelscope.github.io/OpenJudge
• PyPI: py-openjudge