PanCanBench

PanCanBench is a benchmark of 282 de-identified authentic pancreatic cancer patient questions paired with 3,130expert-designed rubrics for evaluating large language models(LLM).

Dataset Structure

Overview

PanCanBench is distributed as a structured JSON dataset. Each entry corresponds to a single real-world patient or caregiver question and contains a detailed, multi-item scoring rubric.

Each question includes:

• question_number (int) — Unique identifier for the question
• question_text (string) — De-identified authentic patient or caregiver question
• rubric_items (list) — A list of structured grading criteria

Each rubric item includes:

• item_number (int) — Unique identifier within the question
• description (string) — Detailed scoring criterion
• min_points (int) — Minimum possible score
• max_points (int) — Maximum possible score

---

Scoring Framework

Each rubric item is scored within a range of −10 to +10 points.

• Positive points reward inclusion of correct, guideline-aligned, and clinically appropriate content.
• Negative points penalize undesirable content, including misinformation, unsafe recommendations, or harmful guidance.

Example Entry

{
  "question_number": 13,
  "question_text": "My loved one has been on chemotherapy for 6 months and the CA 19-9 levels are not going down. Does this mean the treatment isn’t working?",
  "rubric_items": [
    {
      "item_number": 1,
      "description": "The response should state that doctors use imaging results to determine response to therapy.",
      "min_points": 0,
      "max_points": 10
    },
    {
      "item_number": 2,
      "description": "The response should state that CA 19-9 is one part of what determines response to therapy.",
      "min_points": 0,
      "max_points": 10
    },
    {
      "item_number": 3,
      "description": "The response should state that CA 19-9 levels are not always correlated with tumor size or disease status, and note that some patients do not secrete CA 19-9 at all.",
      "min_points": 0,
      "max_points": 10
    }
  ]
}