World-aware Planning Narratives Enhance Large Vision-Language Model Planner

📄 Paper | 🖥️ Code |

Junhao Shi*, Zhaoye Fei*, Siyin Wang, Qipeng Guo, Jingjing Gong, Xipeng Qiu

Fudan University, Shanghai Innovation Institute, Shanghai AI Laboratory

🔥Overview

This repository contains the official implementation of our paper on enhancing large vision-language models (LVLMs) with world-aware planning narratives. Our approach bridges the gap between high-level task instructions and nuanced real-world environments by integrating contextual world knowledge into planning systems.

🚀Key Features

• Framework for enhancing vision-language models with contextual reasoning
• Significant improvements in planning ability, include commonsense reasoning and long-horizon planning tasks
• Outperforms proprietary systems like GPT-4o and Claude-3.5-Sonnet

🔧 Model Settings

Our framework employs Qwen2.5-VL-72B-Instruct as the teacher model for instruction augmentation and reasoning generation. We evaluate our approach on two foundation model series:

• Qwen2.5-VL (Qwen2.5-VL-7B-Instruct)
• InternVL3 (InternVL3-8B)

🚀Performance

Our approach achieves substantial improvements over baseline methods:

• +60.7 absolute improvement in average task success rates with Qwen2.5-VL
• +60.0 in commonsense reasoning
• +70.0 in long-horizon planning

The enhanced open-source models outperform recent proprietary systems by a large margin.

🛠️Evaluation

We evaluate on the EB-ALFRED benchmark from EmbodiedBench, using Success Rate (SR) as the primary metric.

Citation

If you find this work useful for your research, please cite our paper:

@misc{shi2025worldawareplanningnarrativesenhance,
      title={World-aware Planning Narratives Enhance Large Vision-Language Model Planner}, 
      author={Junhao Shi and Zhaoye Fei and Siyin Wang and Qipeng Guo and Jingjing Gong and Xipeng Qiu},
      year={2025},
      eprint={2506.21230},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2506.21230}, 
}