---

📌 Introduction

SPAgent is a foundation agent for perception, reasoning, and action in the physical and spatial world. It provides a modular, open-ended ecosystem of expert tools spanning 2D vision, 3D reconstruction, world modeling, video/image generation, and beyond — enabling grounded spatial reasoning and flexible interaction in complex real-world environments.

📚 Documentation

✅ Features

• Modular Tool System — Mix and match any combination of expert tools
• Dynamic Tool Management — Add/remove tools at runtime
• Parallel Tool Execution — Automatic concurrent processing when possible
• Multi-Image Analysis — Handle single or multiple images seamlessly
• Multiple Model Support — GPT, Qwen, and local VLLM models
• Customizable System Prompt — Per-agent templates; built-in 3D spatial and general vision presets
• Multimodal Agent Memory — AgentMemory records every turn: text, images, tool calls, and results
• Multi-turn Stateful Conversations — Pass AgentMemory across step() calls; save/load sessions
• Reinforcement Learning — GRPO training via ms-swift with multi-turn tool calling and an offline Pi3X cache (no expert servers needed in the training loop) → see RL Training

🛠️ Installation & Setup

1. Environment

conda create -n spagent python=3.11
conda activate spagent

pip install -r requirements.txt
pip install "httpx[socks]"

2. API Keys

# OpenAI (also used by SoraTool)
export OPENAI_API_KEY="your_api_key"
export OPENAI_BASE_URL="your_base_url"

# Qwen / DashScope (apply at: https://bailian.console.aliyun.com)
export DASHSCOPE_API_KEY="your_api_key"

# Moondream (apply at: https://moondream.ai)
export MOONDREAM_API_KEY="your_api_key"

# Google Gemini (used by VeoTool)
export GOOGLE_API_KEY="your_google_api_key"

3. Deploy Expert Servers

See Tool Reference for per-tool deployment instructions (Depth, SAM2, GroundingDINO, Pi3, Molmo2, OrientAnythingV2, WildDet3D, FlowSeek, PaddleOCR-VL, OneFormer, Sana, Qwen Image Edit, VACE, …).

🚀 Quick Start

Basic Usage

from spagent import SPAgent
from spagent.models import GPTModel
from spagent.tools import DepthEstimationTool, SegmentationTool

model = GPTModel(model_name="gpt-4o-mini")
tools = [
    DepthEstimationTool(use_mock=True),
    SegmentationTool(use_mock=True),
]

agent = SPAgent(model=model, tools=tools)
result = agent.solve_problem("image.jpg", "Analyze depth and main objects in this image")
print(result['answer'])

Multi-Tool Agent

from spagent.tools import (
    DepthEstimationTool, SegmentationTool,
    ZoomObjectTool,      # GroundingDINO: crop close-up for attribute inspection
    LocalizeObjectTool,  # GroundingDINO: bbox annotation for spatial/counting
    SupervisionTool, YOLOETool,
)

tools = [
    DepthEstimationTool(use_mock=True),
    SegmentationTool(use_mock=True),
    ZoomObjectTool(use_mock=True),
    LocalizeObjectTool(use_mock=True),
    SupervisionTool(use_mock=True),
    YOLOETool(use_mock=True),
]

agent = SPAgent(model=GPTModel(model_name="gpt-4o-mini"), tools=tools, max_workers=4)
result = agent.solve_problem("image.jpg", "Comprehensively analyze this image")
print(result['answer'])
print(result['used_tools'])

Real Expert Services

tools = [
    DepthEstimationTool(use_mock=False, server_url="http://localhost:20019"),
    SegmentationTool(use_mock=False,    server_url="http://localhost:20020"),
    ZoomObjectTool(use_mock=False,      server_url="http://localhost:20022"),
    LocalizeObjectTool(use_mock=False,  server_url="http://localhost:20022"),
]

For video generation, image generation, multi-turn AgentMemory, custom system prompts, RL training, and detailed testing, see Advanced Examples.

📊 Evaluation

scripts/quick_eval.py is the unified entry point. It runs SPAgent over VLMEvalKit benchmarks and local datasets with automatic resuming and per-sample traces.

60-second smoke test (no servers needed):

python scripts/quick_eval.py --model gpt-4.1-mini --datasets MMStar --limit 5

With tools:

python scripts/quick_eval.py \
    --model gpt-4.1-mini --tools zoom localize \
    --datasets MMStar VStarBench --limit 50 \
    --detection-url http://localhost:20022

Full recipes → REPRODUCE.md · QUICK_EVAL.md · EVALUATION.md

🎯 Reinforcement Learning

SPAgent now supports GRPO reinforcement learning of the policy VLM with multi-turn tool calling, powered by ms-swift. The Pi3X expert runs offline during rollouts: point clouds are pre-computed once and cached to disk, so training needs no live expert servers or network calls.

# 1. Fix dataset image paths
python scripts/build_rl_dataset.py \
    --input "dataset/crossviewQA_train_rl (1).jsonl" \
    --output dataset/crossviewQA_train_rl_fixed.jsonl

# 2. Pre-compute the Pi3X point-cloud cache (run once)
python train/precompute_pi3x_cache.py \
    --dataset dataset/crossviewQA_train_rl_fixed.jsonl \
    --cache-dir dataset/pi3x_cache \
    --checkpoint checkpoints/pi3x/model.safetensors

# 3. Launch GRPO training
cd train && bash train_grpo.sh

Full guide (dataset prep, caching, reward functions, scheduler, post-training) → RL Training

⚠️ Important Notes

• Python Version: Python 3.11 recommended; other versions may have compatibility issues
• GPU Memory: Real mode requires ≥ 24 GB GPU memory
• Concurrency: Control parallel tool execution via max_workers

📝 Citation

@article{zhang2026think3d,
  title={Think3D: Thinking with Space for Spatial Reasoning},
  author={Zhang, Zaibin and Wu, Yuhan and Jia, Lianjie and Wang, Yifan and Zhang, Zhongbo and Li, Yijiang and Ran, Binghao and Zhang, Fuxi and Sun, Zhuohan and Yin, Zhenfei and others},
  journal={arXiv preprint arXiv:2601.13029},
  year={2026}
}

⭐ Star History