Agentic-MME: What Agentic Capability Really Brings to Multimodal Intelligence?

🔬 Process Benchmark · 📊 ACC + Diagnostic Scores

---

📰 News

---

Contents

• Agentic-MME Overview
• Why Agentic-MME Is Different
• Dataset at a Glance
• Dataset Usage
• Environment Setup
• Configuration
• Running Experiments
• Evaluation
• Sharded Runs
• Available Tools
• Output Files
• Project Structure
• Citation

---

Agentic-MME Overview

Agentic-MME is a process-verified benchmark for evaluating multimodal agentic capabilities.

Unlike final-answer-only benchmarks, Agentic-MME explicitly evaluates whether a model can:

1. see better through active visual operations (Visual Expansion), and
2. know better through open-web retrieval (Knowledge Expansion),

while maintaining correct multi-turn reasoning under realistic interaction budgets.

Highlights:

• 🧠 Process-level auditing instead of answer-only grading
• 🖼️ Visual operation correctness and artifact verification
• 🌐 Search strategy and evidence quality verification
• ⏱️ Efficiency diagnosis via overthinking vs. human trajectories

---

Why Agentic-MME Is Different

1) Two core capability dimensions

• 🖼️ Visual Expansion: active image manipulation (crop/rotate/enhance/...) to reveal latent visual cues.
• 🌐 Knowledge Expansion: open-web retrieval (search/lens/webpage reading) to obtain external evidence beyond parametric memory.

2) Progressive task difficulty (not a flat score)

• Level 1: single decisive visual operation.
• Level 2: short multi-step visual + retrieval workflow.
• Level 3: iterative, interleaved visual-retrieval synergy under ambiguity.

3) Process-level dual-axis scoring

V-axis is further decomposed for diagnosis:

• V-tool: whether expected visual operations were invoked.
• V-true: whether produced visual artifacts actually satisfy the checkpoint.

4) Efficiency beyond correctness

• ⏱️ Overthinking is measured relative to human reference trajectories, capturing redundant or excessive tool usage.

---

Dataset at a Glance

Core statistics

Difficulty distribution

Domain distribution (6 major domains)

---

Dataset Usage

🤗 Load from Hugging Face

from datasets import load_dataset

ds = load_dataset("Crystal1047/Agentic-MME", split="train")
print(ds)
print(ds.features)

🗂️ Official dataset folder layout

<dataset_root>/
├── image_cause/
├── images/
├── json/
└── search_url/

Folder purpose:

🧭 Recommended runner invocation with this layout

python general/run_general_script_openai.py \
  --task_dir <dataset_root>/json \
  --dataset_root <dataset_root> \
  --images_dir <dataset_root>/images \
  --model gpt-4o \
  --api_config configs/api.json

Notes:

• --images_dir is optional, but passing it explicitly avoids ambiguity.
• --dataset_root should point to the folder that directly contains json/ and images/.
• If omitted, dataset_root is inferred from the task JSON path.

---

Environment Setup

git clone https://github.com/ChoS3nE11ven/Agentic-MME.git
cd Agentic-MME
conda create -n agenticmme python=3.9 -y
conda activate agenticmme
pip install -r requirements.txt

For local models (Thyme/DeepEyes):

pip install torch transformers accelerate

---

Configuration

🔐 OpenAI API

cp configs/api.json.example configs/api.json

Then edit configs/api.json with your key/base URL.

🌐 Web retrieval config

Edit configs/search_config.json:

• serper_api_key: Serper.dev key for Google Search + Google Lens
• imgbb_api_key: ImgBB key for image upload
• jina_api_key: Jina Reader key for webpage content extraction

---

Running Experiments

🚀 Quick sanity check (single task)

General mode + OpenAI

python general/run_general_script_openai.py \
  --task_json <task_json> \
  --dataset_root <dataset_root> \
  --images_dir <dataset_root>/images \
  --model gpt-4o \
  --api_config configs/api.json \
  --enable_search \
  --search_config configs/search_config.json \
  --max_rounds 15 \
  --max_tool_calls 15

Atomic mode + OpenAI

python atomic/run_atomic_tools_openai.py \
  --task_json <task_json> \
  --dataset_root <dataset_root> \
  --images_dir <dataset_root>/images \
  --model gpt-4o-mini \
  --api_config configs/api.json \
  --enable_search \
  --search_config configs/search_config.json \
  --max_rounds 15 \
  --max_tool_calls 15

📦 Batch run (general mode)

OpenAI

python general/run_general_script_openai.py \
  --task_dir <dataset_root>/json \
  --dataset_root <dataset_root> \
  --images_dir <dataset_root>/images \
  --model gpt-4o \
  --api_config configs/api.json \
  --enable_search \
  --search_config configs/search_config.json \
  --max_rounds 15 \
  --max_tool_calls 15 \
  --skip_existing \
  --shard <shard_idx> \
  --num_shards <num_shards>

Use --num_shards 1 --shard 0 for single-process run.

Local models

# Thyme
python -m general.run_general_script_thyme \
  --task_dir <dataset_root>/json \
  --dataset_root <dataset_root> \
  --images_dir <dataset_root>/images \
  --model_path /path/to/thyme-model \
  --enable_search \
  --search_config configs/search_config.json \
  --skip_existing \
  --shard <shard_idx> \
  --num_shards <num_shards>

# DeepEyes
python -m general.run_general_script_deepeyes \
  --task_dir <dataset_root>/json \
  --dataset_root <dataset_root> \
  --images_dir <dataset_root>/images \
  --model_path /path/to/deepeyes-model \
  --enable_search \
  --search_config configs/search_config.json \
  --skip_existing \
  --shard <shard_idx> \
  --num_shards <num_shards>

⚙️ Batch run (atomic mode)

OpenAI

python atomic/run_atomic_tools_openai.py \
  --task_dir <dataset_root>/json \
  --dataset_root <dataset_root> \
  --images_dir <dataset_root>/images \
  --model gpt-4o-mini \
  --api_config configs/api.json \
  --enable_search \
  --search_config configs/search_config.json \
  --max_rounds 15 \
  --max_tool_calls 15 \
  --skip_existing \
  --shard <shard_idx> \
  --num_shards <num_shards>

Use --num_shards 1 --shard 0 for single-process run.

Local models

# Thyme
python atomic/run_atomic_tools_thyme.py \
  --task_dir <dataset_root>/json \
  --dataset_root <dataset_root> \
  --images_dir <dataset_root>/images \
  --model_path /path/to/thyme-model \
  --enable_search \
  --search_config configs/search_config.json \
  --skip_existing \
  --shard <shard_idx> \
  --num_shards <num_shards>

# DeepEyes
python atomic/run_atomic_tools_deepeyes.py \
  --task_dir <dataset_root>/json \
  --dataset_root <dataset_root> \
  --images_dir <dataset_root>/images \
  --model_path /path/to/deepeyes-model \
  --enable_search \
  --search_config configs/search_config.json \
  --skip_existing \
  --shard <shard_idx> \
  --num_shards <num_shards>

🧩 Important arguments

---

Evaluation

🧭 Evaluation Tracks

We recommend Track A as the default protocol.
At the same time, we also encourage and support Track B when teams use different harnesses or tool-execution stacks, so comparisons are less affected by harness implementation details.

📊 Track A · Step 1: run official scoring

python evaluation/eval_runs_search.py \
  --runs_dir runs/general/gpt-4o \
  --api_config configs/api.json

Default summary output:

runs/scores/general_gpt-4o_scored.json

🧮 Track A · Step 2: aggregate (only if shard eval)

python evaluation/aggregate_shards.py \
  --runs_dir runs/general/gpt-4o

Output:

runs/scores/general_gpt-4o_aggregated.json

🔍 Track A · Step 3: optional V-axis breakdown

python evaluation/analyze_v.py \
  <score_json_path> \
  <dataset_root>/json

<score_json_path> rule (important):

• non-shard run: runs/scores/*_scored.json
• shard run: runs/scores/*_aggregated.json (after Step 2)

This script breaks V-axis into tool-use and visual-check, and reports Overall / L1 / L2 / L3.

🎯 Track B · Task-only ACC (Harness-Friendly)

If you run with a custom harness, we suggest reporting:

• model name + decoding settings
• evaluated split/task count
• final answer ACC only on the released tasks

This keeps comparisons simple and robust when tool sandboxing/execution details differ across harnesses.

---

Sharded Runs

This mode is optional. Use exactly the same run/eval commands as above, and only add:

--shard <shard_idx> --num_shards <num_shards>

Minimal example (shard_idx=0, num_shards=8):

python general/run_general_script_openai.py \
  --task_dir <dataset_root>/json \
  --dataset_root <dataset_root> \
  --images_dir <dataset_root>/images \
  --model gpt-4o \
  --api_config configs/api.json \
  --skip_existing \
  --shard 0 \
  --num_shards 8

Then run the same command with --shard 1, --shard 2, ..., and aggregate with:

python evaluation/aggregate_shards.py --runs_dir runs/general/gpt-4o

---

Available Tools

🖼️ Atomic image tools (visual expansion)

🌐 Retrieval tools (knowledge expansion)

bbox_2d uses normalized [x1, y1, x2, y2] in [0,1000], where (0,0) is top-left and (1000,1000) is bottom-right.

---

Output Files

Each task output is stored under:

runs/{mode}/{model}/{task_id}/

Typical files:

---

Project Structure

Agentic-MME/
├── atomic/
│   ├── run_atomic_tools_openai.py
│   ├── run_atomic_tools_thyme.py
│   └── run_atomic_tools_deepeyes.py
├── general/
│   ├── run_general_script_openai.py
│   ├── run_general_script_thyme.py
│   └── run_general_script_deepeyes.py
├── evaluation/
│   ├── eval_runs_search.py
│   ├── analyze_v.py
│   └── aggregate_shards.py
├── configs/
│   ├── api.json.example
│   └── search_config.json
├── assets/
│   └── case.png
├── common_utils.py
├── dataset_utils.py
├── ast_ops.py
├── atomic_toolbox.py
├── search_toolbox.py
├── search_tools.py
├── verifiers.py
└── requirements.txt

---

Citation

If you find Agentic-MME useful, please cite:

@article{wei2026agentic,
  title={Agentic-MME: What Agentic Capability Really Brings to Multimodal Intelligence?},
  author={Wei, Qianshan and Yang, Yishan and Wang, Siyi and Chen, Jinglin and Wang, Binyu and Wang, Jiaming and Chen, Shuang and Li, Zechen and Shi, Yang and Tang, Yuqi and others},
  journal={arXiv preprint arXiv:2604.03016},
  year={2026}
}