Parameter-Inverted Image Pyramid Networks (PIIP)

[📚 PIIP v1 Paper] [📚 PIIP v2 Paper] [🤗 Model] [📝 中文解读] [📑 Slides] [▶️ Video]

The official implementation of Parameter-Inverted Image Pyramid Networks.

NeurIPS 2024 Spotlight & TPAMI 2025

Ranked Top 10 in NeurIPS 2024 (among 15671 submissions), Top 2 in Computer Vision Area

📰 News

[2025/7/25] PIIP-LLaVA is accepted by TPAMI 2025!

[2025/4/20] Code and models of PIIP-LLaVA and ConvNeXt-based detection models are released!

[2025/1/15] We introduce PIIP-LLaVA, an MLLM that uses PIIP design to improve performance and save computational costs. We also extend PIIP to CNN-based structures and ViT-CNN hybrid structures. Code and models will be released soon. Check out our new paper for details.

⭐️ Highlights

TL;DR: We introduce the Parameter-Inverted Image Pyramid Networks (PIIP), employing a parameter-inverted paradigm that uses models with different parameter sizes to process different resolution levels of the image pyramid, thereby saving computation cost while improving the performance.

• Support tasks of object detection, instance segmentation, semantic segmentation , image classification, and multimodal understanding.
• Surpasses single-branch and other multi-resolution methods with higher performance and lower computation costs.
• Achieve 60.0 $\rm AP^b$) on COCO object detection with InternViT-6B, and 73.0% accuracy on TextVQA and 74.5% on MMBench with only 2.8M training data.

🖼 Qualitative Results

📌 Abstract

Image pyramids are widely adopted in top-performing methods to obtain multi-scale features for precise visual perception and understanding. However, current image pyramids use the same large-scale model to process multiple resolutions of images, leading to significant computational cost. To address this challenge, we propose a novel network architecture, called Parameter-Inverted Image Pyramid Networks (PIIP). Specifically, PIIP uses pretrained models (ViTs or CNNs) as branches to process multi-scale images, where images of higher resolutions are processed by smaller network branches to balance computational cost and performance. To integrate information from different spatial scales, we further propose a novel cross-branch feature interaction mechanism. To validate PIIP, we apply it to various perception models and a representative multimodal large language model called LLaVA, and conduct extensive experiments on various tasks such as object detection, segmentation, image classification and multimodal understanding. PIIP achieves superior performance compared to single-branch and existing multi-resolution approaches with lower computational cost. When applied to InternViT-6B, a large-scale vision foundation model, PIIP can improve its performance by 1%-2% on detection and segmentation with only 40%-60% of the original computation, finally achieving 60.0 box AP on MS COCO and 59.7 mIoU on ADE20K. For multimodal understanding, our PIIP-LLaVA achieves 73.0% accuracy on TextVQA and 74.5% on MMBench with only 2.8M training data.

🔍 Method

🛠️ Usage

For instructions on installation, pretrained models, training and evaluation, please refer to the readme files under each subfolder:

• Object detection and instance segmentation: mmdetection
• Semantic segmentation: mmsegmentation
• Image classification: classification
• Multimodal understanding: llava

🚀 Released Models

COCO Object Detection and Instance Segmentation

[!Note]

1. We report the number of parameters and FLOPs of the backbone.
2. Results in the paper were obtained with an internal codebase, which may exhibit slightly different performance than this repo ($\leq\pm0.2$).
3. Experiments involving InternViT-6B do not use window attention, different from those in the paper.

DeiT-based Models (Mask-RCNN, 1x Schedule)

More ViT Variants

ConvNeXt-based Models (Mask-RCNN, 1x Schedule)

C: ConvNeXt. V: ViT (DeiT).

ADE20K Semantic Segmentation

ImageNet-1K Image Classification

Multimodal Understanding

🖊️ Citation

If you find this work helpful for your research, please consider giving this repo a star ⭐ and citing our paper:

@inproceedings{piip,
  title={Parameter-Inverted Image Pyramid Networks},
  author={Zhu, Xizhou and Yang, Xue and Wang, Zhaokai and Li, Hao and Dou, Wenhan and Ge, Junqi and Lu, Lewei and Qiao, Yu and Dai, Jifeng},
  booktitle={NeurIPS},
  year={2024}
}

@article{piip_v2,
  title={Parameter-Inverted Image Pyramid Networks for Visual Perception and Multimodal Understanding},
  author={Wang, Zhaokai and Zhu, Xizhou and Yang, Xue and Luo, Gen and Li, Hao and Tian, Changyao and Dou, Wenhan and Ge, Junqi and Lu, Lewei and Qiao, Yu and Dai, Jifeng},
  journal={IEEE transactions on pattern analysis and machine intelligence},
  year={2025},
  publisher={IEEE}
}

📃 License

This project is released under the MIT license. Parts of this project contain code and models from other sources, which are subject to their respective licenses.

🙏 Acknowledgements

Our code is built with reference to the code of the following projects: LLaVA-1.5, InternVL-MMDetSeg, ViT-Adapter, DeiT, MMDetection, MMSegmentation, and timm. Thanks for their awesome work!