README
PerceptionDLM
PerceptionDLM is a multimodal diffusion language model optimized for efficient parallel region perception. Built upon PerceptionDLM-Base, it fully leverages the parallel decoding nature of diffusion language models (DLMs): given an image and multiple region masks, it generates descriptions for all regions simultaneously within a single denoising process — avoiding the linear latency growth of autoregressive (AR) region captioners.
To the best of our knowledge, this is the first model to achieve parallel region captioning and perception by leveraging the advantages of diffusion language models.
📄 Paper | 💻 Code | 📊 ParaDLC-Bench
Highlights
- 🧩 Parallel region captioning. Region prompting + structured attention masking describe many masked regions in a single denoising pass.
- ⚡ Up to 3.44× throughput speedup in dense multi-region scenarios, with stable per-image latency (~2.9s).
- 🎯 Competitive quality with strong AR region captioners while being substantially faster.
Model Details
| Base model | MSALab/PerceptionDLM-Base |
| Key modules | Region prompting, RoI-aligned feature replay, structured attention masking |
| Region prompts | up to 6 per image |
| Default inference | 32 diffusion steps, generation length 32 per mask |
| Training | full ParaCaption corpus, ~2 days on 32× H100 |
| Precision | bfloat16 |
Results (ParaDLC-Bench)
| Method | Type | Avg (%) | TPF ↑ | Time (s) ↓ |
|---|---|---|---|---|
| GAR-8B | AR (sequential) | 69.5 | 1.0 | 479 |
| LLaDA-V-8B | Diffusion | 35.2 | 1.0 | 3241 |
| PerceptionDLM | Diffusion (parallel) | 62.4 | 2.9 | 276 |
TPF = Tokens Per Forward (higher = more parallel). PerceptionDLM nearly doubles the accuracy of prior diffusion VLMs while drastically reducing inference time.
Usage
Full inference scripts are provided in the GitHub repository.
python demo/infer_pdmllm.py \
--model-path MSALab/PerceptionDLM \
--image assets/demo.jpg \
--masks assets/demo_mask_0.jpg \
assets/demo_mask_1.jpg \
assets/demo_mask_2.jpg \
--gen-length 32 --steps 32 --temperature 0.0 --top-p 1.0
The model takes an RGB image plus one or more binary masks, and returns one caption per region — all generated in parallel.
Citation
@article{sun2026perceptiondlm,
title = {PerceptionDLM: Parallel Region Perception with Multimodal Diffusion Language Models},
author = {Sun, Yueyi and Wang, Yuhao and Li, Jason and Tian, Ye and Zhang, Tao and Mai, Jacky and Wang, Yihan and Wang, Haochen and Bai, Jinbin and Yang, Ling and Tong, Yunhai},
journal = {arXiv preprint arXiv:2606.19534},
year = {2026}
}
License
Released under the Apache License 2.0.
Similar repos
README
PerceptionDLM
PerceptionDLM is a multimodal diffusion language model optimized for efficient parallel region perception. Built upon PerceptionDLM-Base, it fully leverages the parallel decoding nature of diffusion language models (DLMs): given an image and multiple region masks, it generates descriptions for all regions simultaneously within a single denoising process — avoiding the linear latency growth of autoregressive (AR) region captioners.
To the best of our knowledge, this is the first model to achieve parallel region captioning and perception by leveraging the advantages of diffusion language models.
📄 Paper | 💻 Code | 📊 ParaDLC-Bench
Highlights
- 🧩 Parallel region captioning. Region prompting + structured attention masking describe many masked regions in a single denoising pass.
- ⚡ Up to 3.44× throughput speedup in dense multi-region scenarios, with stable per-image latency (~2.9s).
- 🎯 Competitive quality with strong AR region captioners while being substantially faster.
Model Details
| Base model | MSALab/PerceptionDLM-Base |
| Key modules | Region prompting, RoI-aligned feature replay, structured attention masking |
| Region prompts | up to 6 per image |
| Default inference | 32 diffusion steps, generation length 32 per mask |
| Training | full ParaCaption corpus, ~2 days on 32× H100 |
| Precision | bfloat16 |
Results (ParaDLC-Bench)
| Method | Type | Avg (%) | TPF ↑ | Time (s) ↓ |
|---|---|---|---|---|
| GAR-8B | AR (sequential) | 69.5 | 1.0 | 479 |
| LLaDA-V-8B | Diffusion | 35.2 | 1.0 | 3241 |
| PerceptionDLM | Diffusion (parallel) | 62.4 | 2.9 | 276 |
TPF = Tokens Per Forward (higher = more parallel). PerceptionDLM nearly doubles the accuracy of prior diffusion VLMs while drastically reducing inference time.
Usage
Full inference scripts are provided in the GitHub repository.
python demo/infer_pdmllm.py \
--model-path MSALab/PerceptionDLM \
--image assets/demo.jpg \
--masks assets/demo_mask_0.jpg \
assets/demo_mask_1.jpg \
assets/demo_mask_2.jpg \
--gen-length 32 --steps 32 --temperature 0.0 --top-p 1.0
The model takes an RGB image plus one or more binary masks, and returns one caption per region — all generated in parallel.
Citation
@article{sun2026perceptiondlm,
title = {PerceptionDLM: Parallel Region Perception with Multimodal Diffusion Language Models},
author = {Sun, Yueyi and Wang, Yuhao and Li, Jason and Tian, Ye and Zhang, Tao and Mai, Jacky and Wang, Yihan and Wang, Haochen and Bai, Jinbin and Yang, Ling and Tong, Yunhai},
journal = {arXiv preprint arXiv:2606.19534},
year = {2026}
}
License
Released under the Apache License 2.0.