🥥 COCONut-PanCap

COCONut-PanCap is a panoptic-grounded caption dataset for fine-grained visual understanding and generation.

It augments COCO-style panoptic segmentation with region-level natural language descriptions, enabling structured vision-language modeling and evaluation.

---

📌 Dataset Summary

COCONut-PanCap extends panoptic segmentation by associating each segment with descriptive text grounded to that region.

Each sample includes:

• RGB image
• Panoptic segmentation mask
• Region-level grounded captions
• Category metadata

The dataset supports both discriminative and generative multimodal tasks.

---

🗂 Dataset Structure

coconut_pancap/
│
├── caption_train2017.tar
├── pancap_val.tar
├── coconut_pancap_sample.zip
│
├── 000000000142.txt        # Example annotation
├── demo.jpg                # Example image
│
└── vis_pancap.py           # Visualization script

---

🧾 Annotation Format

Each annotation file (e.g., 000000000142.txt) follows a COCO-style structure:

{
  "image_id": 142,
  "file_name": "000000000142.jpg",
  "segments_info": [
    {
      "segment_id": 6,
      "category": "person",
      "caption": "A man wearing a blue shirt standing near a table."
    },
    {
      "segment_id": 12,
      "category": "dog",
      "caption": "A small brown dog sitting on the grass."
    }
  ]
}

---

🖼 Example

See demo.jpg for a sample image and 000000000142.txt for its corresponding annotation.

---

🎯 Supported Tasks

• Panoptic segmetation-grounded captioning
• Fine-grained visual description
• Vision-language evaluation
• Text-to-image benchmarking
• Region-level multimodal reasoning

---

📊 Dataset Statistics

Full statistics are provided in the associated publication.

---

🚀 Usage

Load via Hugging Face Datasets

from datasets import load_dataset

dataset = load_dataset("xdeng77/coconut_pancap")
print(dataset)

---

Visualization

python vis_pancap.py     --ann_json path/to/annotation.json     --img_dir path/to/images     --mask_dir path/to/masks     --out_dir ./vis_out

---

📝 Citation

If you use this dataset, please cite:

@article{deng2025coconut,
  title={Coconut-pancap: Joint panoptic segmentation and grounded captions for fine-grained understanding and generation},
  author={Deng, Xueqing and Yu, Qihang and Athar, Ali and Yang, Chenglin and Yang, Linjie and Jin, Xiaojie and Shen, Xiaohui and Chen, Liang-Chieh},
  journal={arXiv preprint arXiv:2502.02589},
  year={2025}
}

---

⚖ License

This dataset builds upon MS COCO and follows the COCO license terms.

---

🤝 Contact

Xueqing Deng
xueqingdeng77@gmail.com ByteDance Seed Research