ParaDLC-Bench

ParaDLC-Bench (Parallel Detailed Localized Captioning Benchmark) is a benchmark for multi-region localized captioning that jointly evaluates caption quality and inference efficiency. It extends DLC-Bench from single-region evaluation to concurrent multi-region evaluation, explicitly stressing a model's ability to describe many regions at once without cross-region interference.

📄 Paper  |  💻 Code  |  🤖 PerceptionDLM

Key Features

• 🧩 Multi-target focus. Every image contains ≥ 2 masked regions (mostly 2–4, up to 8), selected to be spatially adjacent, semantically confusing, or prone to feature entanglement.
• 🔍 Fine-grained. Intentionally retains many tiny regions; the average mask area ratio is only 0.07.
• 🧪 Reference-free LLM-as-judge. Inherits the DLC-Bench paradigm and extends it from intra-region detail accuracy to inter-region anti-interference (cross-region hallucination) evaluation.

Statistics

• Questions: 2,345 manually verified multiple-choice questions (positive + negative/interference).
• Average mask area ratio: 0.07.

Evaluation Protocol

The evaluation is a two-step, reference-free process:

1. Generation. The model describes all specified masked regions of an image (in parallel if supported, otherwise sequentially).
2. Judging. An LLM judge (GPT-5.2 by default, at temperature=0) scores each caption against predefined questions:
   • Positive questions reward accurate attributes and penalize factual errors.
   • Negative / interference questions penalize hallucinated typical-but-absent parts and, crucially, attributes wrongly borrowed from other concurrent targets in the same image.

Scores are averaged per mask to ensure equal weighting across regions. The benchmark is robust to the choice of judge (verified across GPT-5.2, Gemini-3.1-Pro, and Qwen3.5-27B).

Files

annotations/
├── annotations.json   # images + region masks
├── qa.json            # positive / negative questions per mask
└── class_names.json   # target category names

Leaderboard (excerpt)

Usage

See the Evaluation Guide for the full inference + judging pipeline.

python evaluation/ParaDLC-Bench/infer_mask_captions_paradlc.py \
    --model-path MSALab/PerceptionDLM \
    --image-root annotations/images \
    --anno-json annotations/annotations.json \
    --qa-json annotations/qa.json \
    --gen-length 32 --steps 32 --temperature 0.0 --top-p 1.0

Citation

@article{sun2026perceptiondlm,
  title   = {PerceptionDLM: Parallel Region Perception with Multimodal Diffusion Language Models},
  author  = {Sun, Yueyi and Wang, Yuhao and Li, Jason and Tian, Ye and Zhang, Tao and Mai, Jacky and Wang, Yihan and Wang, Haochen and Bai, Jinbin and Yang, Ling and Tong, Yunhai},
  journal = {arXiv preprint arXiv:2606.19534},
  year    = {2026}
}

License

Released under the Apache License 2.0. Source images originate from Objects365 and DaTaSeg; please also respect their original licenses and terms of use.