MedSimp-JudgeBench

A perturbation-based calibration benchmark for LLM-as-judge medical faithfulness evaluation.

Description

708 reference simplifications with programmatically injected known errors, used to measure sensitivity and specificity of dual LLM judges (Llama-3.3-70B and Qwen3-32B) via Nebius Token Factory.

Error Types

Fields

• idx: sample index from GuyDor007/medisimplifier-dataset test split
• error_type: dose / lateral / negation / diagnosis / none (clean)
• condition: corrupted / clean
• input: original medical discharge summary
• clean_ref: reference simplification (Claude Opus 4.5)
• perturbed: simplification with injected error (or same as clean_ref for controls)
• llama_clean: Llama-3.3-70B verdict on clean reference
• qwen_clean: Qwen3-32B verdict on clean reference
• llama_verdict: Llama-3.3-70B verdict on perturbed/clean text
• qwen_verdict: Qwen3-32B verdict on perturbed/clean text

Usage

Use this benchmark to calibrate your own LLM judge on medical text simplification. Judges run via Nebius Token Factory.

Citation

Part of MediSimplifier - Nebius Serverless AI Builders Challenge submission. GitHub: https://github.com/deepset01-sys/medisimplifier-nebius