🧠🤖 Awesome-Think-With-Images

Introduction

Welcome to Awesome-Think-With-Images! The field of multimodal AI is undergoing a fundamental evolution, moving beyond static visual perception towards a new paradigm where vision becomes a dynamic, manipulable cognitive workspace. This repository is the first comprehensive resource that systematically curates the pivotal research enabling this shift.

We structure this collection along a trajectory of increasing cognitive autonomy, as detailed in our survey. This journey unfolds across three key stages:

1. Stage 1: Tool-Driven Visual Exploration — Models as "Commanders" orchestrating external visual tools.
2. Stage 2: Programmatic Visual Manipulation — Models as "Visual Programmers" creating bespoke analyses.
3. Stage 3: Intrinsic Visual Imagination — Models as "Visual Thinkers" generating internal mental imagery.

As detailed in our survey, this paradigm shift unlocks three key capabilities: Dynamic Perceptual Exploration, Structured Visual Reasoning, and Goal-Oriented Generative Planning. This collection is for researchers, developers, and enthusiasts eager to explore the forefront of AI that can truly see, reason, and imagine.

We structure this collection along a trajectory of increasing cognitive autonomy. This journey unfolds across three key stages, forming the taxonomy of our work:

This collection is for researchers, developers, and enthusiasts eager to explore the forefront of AI that can truly see, reason, and imagine.

---

🔔 News

• [2026-02] We released AgentVista, the first benchmark for multimodal agents on realistic, ultra-challenging visual scenarios requiring long-horizon hybrid tool use.
• [2026-01] Our survey Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers reached 100 citations, and we also updated the paper list with newly released papers in this rapidly evolving area.
• [2025-10] We reached 1,000 stars on this repository. Thanks for the support from the community!
• [2025-07] We have released "Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers", the first comprehensive survey dedicated to the emerging paradigm of "Think with Images".
• [2025-06] We created this repository to maintain a paper list on Awesome-Think-With-Images. Contributions are welcome!
• [2025-05] We are excited to release OpenThinkIMG, the first dedicated end-to-end open-source framework designed to empower LVLMs to truly think with images! For ease of use, we've configured a Docker environment. We warmly invite the community to explore, use, and contribute.

---

📜 Table of Contents

• Stage 1: Tool-Driven Visual Exploration
  • Prompt-Based Approaches
  • SFT-Based Approaches
  • RL-Based Approaches
• Stage 2: Programmatic Visual Manipulation
  • Prompt-Based Approaches
  • SFT-Based Approaches
  • RL-Based Approaches
• Stage 3: Intrinsic Visual Imagination
  • SFT-Based Approaches
  • RL-Based Approaches
• Evaluation & Benchmarks
  • Benchmarks for Thinking with Images
• Contributing & Citation

---

🧭 The Three-Stage Evolution of Thinking with Images

This section provides a conceptual map to navigate the paper list. The following papers are organized according to the primary mechanism they employ, aligning with the three-stage framework from our survey.

---

🛠️ Stage 1: Tool-Driven Visual Exploration

In this stage, the model acts as a planner, orchestrating a predefined suite of external visual tools. Intelligence is demonstrated by selecting the right tool for the right sub-task.

➤ Prompt-Based Approaches

Leveraging in-context learning to guide tool use without parameter updates.

• Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language
• PromptCap: Prompt-Guided Task-Aware Image Captioning
• MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action
• What does CLIP know about a red circle? Visual prompt engineering for VLMs
• Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V
• DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLM
• Chain-of-Spot: Interactive Reasoning Improves Large Vision-Language Models
• Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models
• ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Exploration
• DyFo: A Training-Free Dynamic Focus Visual Search for Enhancing LMMs in Fine-Grained Visual Understanding
• VisuoThink: Empowering LVLM Reasoning with Multimodal Tree Search
• Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought
• Visual Abstract Thinking Empowers Multimodal Reasoning
• MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs
• InSight-o3: Empowering Multimodal Foundation Models with Generalized Visual Search

➤ SFT-Based Approaches

Fine-tuning models on data demonstrating how to invoke tools and integrate their outputs.

• LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents
• V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs
• CogCoM: Train Large Vision-Language Models Diving into Details through Chain of Manipulations
• Visual CoT: Unleashing Chain-of-Thought Reasoning in Multi-Modal Language Models
• Instruction-Guided Visual Masking
• From the Least to the Most: Building a Plug-and-Play Visual Reasoner via Data Synthesis
• TACO: Learning Multi-modal Action Models with Synthetic Chains-of-Thought-and-Action
• CMMCoT: Enhancing Complex Multi-Image Comprehension via Multi-Modal Chain-of-Thought and Memory Augmentation
• UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learning
• Don’t Look Only Once: Towards Multimodal Interactive Reasoning with Selective Visual Revisitation
• VGR: Visual Grounded Reasoning
• Multi-Step Visual Reasoning with Visual Tokens Scaling and Verification
• WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agent
• SceneCOT: Eliciting Grounded Chain-of-Thought Reasoning in 3D Scenes

➤ RL-Based Approaches

Using rewards to train agents to discover optimal tool-use strategies.

• Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcement
• [DWIM: Towards Tool-aware Visual Reasoning via Discrepancy-aware Workflow Generation & Instruct-Masking Tuning] (https://arxiv.org/pdf/2503.19263) ![https://img.shields.io/badge/abs-2025.03-red]
• GRIT: Teaching MLLMs to Think with Images
• Point-RFT: Improving Multimodal Reasoning with Visually Grounded Reinforcement Finetuning
• OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learning
• VisionReasoner: Unified Visual Perception and Reasoning via Reinforcement Learning
• DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning
• VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use
• Chain-of-Focus: Adaptive Visual Search and Zooming for Multimodal Reasoning via RL
• VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use![https://img.shields.io/badge/abs-2025.05-red]
• Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learning
• One RL to See Them All: Visual Triple Unified Reinforcement Learning
• UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learning
• VisualToolAgent (VisTA): A Reinforcement Learning Framework for Visual Tool Selection
• Active-O3: Empowering Multimodal Large Language Models with Active Perception via GRPO
• Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawing
• VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learning
• WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agent
• VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use ![https://img.shields.io/badge/abs-2025.09-red]
• Thinking with Images via Self-Calling Agent ![https://img.shields.io/badge/abs-2025.12-red]

---

💻 Stage 2: Programmatic Visual Manipulation

Here, models evolve into "visual programmers," generating executable code (e.g., Python) to create custom visual analyses. This unlocks compositional flexibility and interpretability.

➤ Prompt-Based Approaches

Guiding models to generate code as a transparent, intermediate reasoning step.

• Visual programming: Compositional visual reasoning without training
• ViperGPT: Visual Inference via Python Execution for Reasoning
• Visual sketchpad: Sketching as a visual chain of thought for multimodal language models
• VipAct: Visual-perception enhancement via specialized vlm agent collaboration and tool-use
• SketchAgent: Language-Driven Sequential Sketch Generation
• CAD-Assistant: Tool-Augmented VLLMs as Generic CAD Task Solvers?
• MMFactory: A Universal Solution Search Engine for Vision-Language Tasks
• ReFocus: Visual Editing as a Chain of Thought for Structured Image Understanding
• Interactive Sketchpad: A Multimodal Tutoring System for Collaborative, Visual Problem-Solving
• Agentic Vision with Dynamic Tooling

➤ SFT-Based Approaches

Distilling programmatic logic into models or using code to bootstrap high-quality training data.

• Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Models
• ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Models
• Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generation
• Advancing vision-language models in front-end development via data synthesis
• COMPACT: COMPositional Atomic-to-Complex Visual Capability Tuning
• MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal Mathematical Reasoning
• PyVision: Agentic Vision with Dynamic Tooling
• Thyme: Think Beyond Images

➤ RL-Based Approaches

Optimizing code generation policies using feedback from execution results.

• Visual Agentic Reinforcement Fine-Tuning
• ProgRM: Build Better GUI Agents with Progress Rewards
• Thyme: Think Beyond Images

---

🎨 Stage 3: Intrinsic Visual Imagination

The most advanced stage, where models achieve full cognitive autonomy. They generate new images or visual representations internally as integral steps in a closed-loop thought process.

➤ SFT-Based Approaches

Training on interleaved text-image data to teach models the grammar of multimodal thought.

• Generating images with multimodal language models
• NExT-GPT: Any-to-Any Multimodal LLM
• Minigpt-5: Interleaved vision-and-language generation via generative vokens
• Generative multimodal models are in-context learners
• SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation
• Chameleon: Mixed-Modal Early-Fusion Foundation Models
• Show-o: One single transformer to unify multimodal understanding and generation
• Metamorph: Multimodal understanding and generation via instruction tuning
• LMFusion: Adapting Pretrained Language Models for Multimodal Generation
• TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation
• Imagine while Reasoning in Space: Multimodal Visualization-of-Thought
• GoT: Unleashing Reasoning Capability of Multimodal Large Language Model for Visual Generation and Editing
• Cot-vla: Visual chain-of-thought reasoning for vision-language-action models
• Transfer between Modalities with MetaQueries
• BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset
• Emerging properties in unified multimodal pretraining
• Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generation
• Thinking with Generated Images
• UniFork: Exploring Modality Alignment for Unified Multimodal Understanding and Generation
• Show-o2: Improved Native Unified Multimodal Models
• MathCanvas: Intrinsic Visual Chain-of-Thought for Multimodal Mathematical Reasoning

➤ RL-Based Approaches

Empowering models to discover generative reasoning strategies through trial, error, and reward.

• Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Step
• Visual Planning: Let's Think Only with Images
• T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoT
• MindOmni: Unleashing Reasoning Generation in Vision Language Models with RGPO
• GoT-R1: Unleashing Reasoning Capability of MLLM for Visual Generation with Reinforcement Learning
• Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPO
• Robotic Control via Embodied Chain-of-Thought Reasoning
• FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving
• ControlThinker: Unveiling Latent Semantics for Controllable Image Generation through Visual Reasoning
• Qwen-Image Technical Report

---

📊 Evaluation & Benchmarks

Essential resources for measuring progress. These benchmarks are specifically designed to test the multi-step, constructive, and simulative reasoning capabilities required for "Thinking with Images".

➤ Benchmarks for Thinking with Images

• A Cognitive Evaluation Benchmark of Image Reasoning and Description for Large Vision-Language Models
• m&m's: A Benchmark to Evaluate Tool-Use for multi-step multi-modal Tasks
• M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal
• Vgbench: Evaluating large language models on vector graphics understanding and generation
• ARC Prize 2024: Technical Report
• CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Models
• CrossWordBench: Evaluating the Reasoning Capabilities of LLMs and LVLMs with Controllable Puzzle Generation
• WorldScore: A Unified Evaluation Benchmark for World Generation
• MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models
• PointArena: Probing Multimodal Grounding Through Language-Guided Pointing
• ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models
• Can MLLMs Guide Me Home? A Benchmark Study on Fine-Grained Visual Reasoning from Transit Maps
• PhyX: Does Your Model Have the "Wits" for Physical Reasoning?
• OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoning
• VisualQuality-R1: Reasoning-Induced Image Quality Assessment via Reinforcement Learning to Rank
• GEMeX-ThinkVG: Towards Thinking with Visual Grounding in Medical VQA via Reinforcement Learning
• ViC-Bench: Benchmarking Visual-Interleaved Chain-of-Thought Capability in MLLMs with Free-Style Intermediate State Representations
• TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoning

---

🙏 Contributing & Citation

We welcome contributions! If you have a paper that fits into this framework, please open a pull request. Let's build this resource together.

If you find our survey and this repository useful for your research, please consider citing our work:

@article{su2025thinking,
  title={Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers},
  author={Su, Zhaochen and Xia, Peng and Guo, Hangyu and Liu, Zhenhua and Ma, Yan and Qu, Xiaoye and Liu, Jiaqi and Li, Yanshu and Zeng, Kaide and Yang, Zhengyuan and others},
  journal={arXiv preprint arXiv:2506.23918},
  year={2025}
}

Star History