[ACM MM 2026] Clarify Before Executing: A Self-Evolving Agent for Resolving Intent Asymmetry in 3D Tool Orchestration

Xiaoye Zhu, Weixin Li, Junan Huo, Bozhong Wang, Jia Zeng, Yi Yang, Cen Chen, Qi Liu^†

South China University of Technology
^†Corresponding author

Paper | Demo | Code | Benchmark | Dataset Assets | Reproduction | 中文

CLARE places a clarification gate before 3D tool execution, resolving ambiguity, missing information, and mistaken constraints through targeted dialogue. This release centers on the 3D-Clarify benchmark and a reproducible code-level evaluation pipeline for the paper's single-step and multi-step experiments.

Interactive Demo

Open the CLARE interactive trajectory demo to replay ambiguity, missing-information, and conflicting-constraint cases in the browser. The demo is a self-contained static site under demo/ and does not invoke a model or 3D executor.

Highlights

• Clarify before executing: four specialized components gate execution, maintain clarified state, synthesize 3D tool code, and communicate targeted feedback.
• 3D-Clarify benchmark: 620 tasks across five 3D domains, each evaluated under ambiguity, missing-information, and mistaken-detail conditions.
• Self-evolving clarification: simulated multi-turn interaction and Multi-turn Reward optimization raise task success to 60.40% on Single and 43.34% on Multi.

Selected paper results report average soft CR/SR (%) across all three intent-asymmetry conditions:

Start Here

3D-Clarify Benchmark

The same benchmark files are hosted as the xyzhu1225/3D-Clarify dataset and can be loaded with load_dataset("xyzhu1225/3D-Clarify").

Each task provides one fully specified hidden goal and three controlled intent-asymmetry conditions:

The paper evaluates 500 x 3 = 1,500 single-step and 120 x 3 = 360 multi-step task-condition instances. The release includes fixed ground-truth Executor calls for deterministic CR/SR evaluation. See the dataset guide for domain distributions, mistaken-detail categories, record fields, and SHA-256 hashes.

Optional Source Assets

Code-level paper evaluation does not execute 3D tools and does not require source media. Physical execution requires the separately hosted CLARE-assets-v1.zip archive (427 files, 443 MiB compressed). Download it from the CLARE-assets-v1 dataset on Hugging Face, then extract it at the repository root:

unzip /path/to/CLARE-assets-v1.zip -d .
echo "77b5b068f5105e0c95f14d94cd458da52e2187042c8c301e6e7f9076a147fcf9  /path/to/CLARE-assets-v1.zip" | sha256sum -c -

This creates data/source/, matching the benchmark's released input paths.

Reproduce The Main Table

The provided launcher uses four persistent vLLM services: two base-model replicas plus one merged SFT Approver and one merged DPO Approver. The default layout uses four 24 GB GPUs. Main-table clients load only the tokenizer; physical 3D tools remain disabled.

1. Install The Client Environment

git clone https://github.com/xyzhu1225/CLARE.git
cd CLARE
conda create -n clare python=3.10 -y
conda activate clare
pip install torch --index-url https://download.pytorch.org/whl/cu124
pip install -e '.[dev]'

Install vLLM in a compatible environment and identify its executable. Prepare the Qwen3-8B base model and the two LoRA Approver checkpoints.

2. Configure Model Paths

export PYTHON="$(command -v python)"
export VLLM_BIN=/path/to/vllm-env/bin/vllm
export BASE_MODEL=/path/to/Qwen3-8B
export MODEL_PATH="$BASE_MODEL"
export SFT_ADAPTER=/path/to/clare-se-sft
export DPO_ADAPTER=/path/to/clare-se-dpo
export SFT_MODEL_PATH="$SFT_ADAPTER"
export DPO_MODEL_PATH="$DPO_ADAPTER"
export MERGED_ROOT=/path/to/CLARE-merged

3. Merge, Smoke-Test, And Run

bash scripts/vllm/merge_checkpoints.sh
bash scripts/vllm/start_servers.sh
bash scripts/vllm/run_smoke.sh
bash scripts/vllm/run_full.sh

The full run is launched in resumable tmux sessions. Monitor progress and stop the model services with:

bash scripts/vllm/status.sh
bash scripts/vllm/stop_servers.sh

A complete run contains 1,500 Single and 360 Multi result files per method. Predictions, logs, and metrics.json files are written under outputs/vllm_full/; paper_comparison.md is generated after all methods finish. Full environment variables, individual method commands, resume behavior, sharding, evaluation details, and ablations are documented in Reproduction.

Evaluation Metrics

The evaluator reports paper-aligned soft metrics and their strict counterparts:

• CR: mean correctness of ground-truth tool arguments across calls and tasks.
• SR: percentage of tasks whose complete tool-call sequence is correct.
• Soft: prompt-semantic equivalence plus the documented path-only rescue.
• Strict: structured tool/argument matching without path-only rescue.

Evaluation is read-only and never inserts ground-truth code into prediction files. The exact scoring contract and standalone evaluator command are in Reproduction.

vLLM Sampling Note

The paper's original results used Hugging Face Transformers. This release uses vLLM for throughput, so stochastic trajectories are not expected to be token-for-token identical. The difference is most visible for iterative baselines whose sampled actions become later context. Values below are soft CR/SR (%):

Differences for the remaining methods are smaller and do not change the main method ordering or conclusions.

Self-Evolution And Training

Generate trajectories, then build the SFT and DPO records:

MODEL_PATH=/path/to/Qwen3-8B \
  bash scripts/training/sample_trajectories.sh
bash scripts/training/build_sft_data.sh
bash scripts/training/build_dpo_data.sh

Ready-to-use LLaMA-Factory LoRA configurations are provided for SFT and DPO. Register the generated JSON as described in the YAML comments, update model/output paths, and follow the training instructions.

Repository Layout

assets/                paper teaser used by the repository overview
clare/                 agents, baselines, evaluation, and training utilities
configs/               tool and capability registry
data/                  released benchmark and few-shot examples
results/               paper aggregate results
scripts/main_table/    one entrypoint per main-table method
scripts/vllm/          service, smoke, full-run, and monitoring utilities
scripts/training/      self-evolution builders and LLaMA-Factory configs
docs/                  dataset and reproduction guides
tests/                 model-free contract tests
tools/                 optional pinned 3D tool submodules

License

The CLARE orchestration code is released under the MIT License. Repositories under tools/ retain their upstream licenses. The separately hosted benchmark assets retain their original source licenses; model checkpoints are not redistributed here.