Actor as Its Own Critic: Unifying Region Understanding and Localization via CycleGRPO

Official implementation of CycleGRPO for jointly improving region captioning and referring localization with multimodal reinforcement learning. CycleGRPO uses a caption ↔ grounding cycle-consistency reward. The policy is a Qwen3-VL-4B that emits mask tokens decoded by a VQ-SAM2 mask tokenizer; the reward runs an inner grounding rollout conditioned on the model's own caption and scores it by mask IoU — so captions are optimized to be distinctive and locatable, with no caption ground-truth needed in the RL stage.

Built on EasyR1 / veRL (see README_EasyR1.md for the underlying framework).

Repo layout

verl/                     # RL engine (forked EasyR1/veRL): trainer, FSDP workers, vLLM rollout
projects/
  rl/
    qwen3vl_4b_mt.sh       # >>> main image CycleGRPO training entry <<<
    config.yaml            # default RL config (algorithm, rollout, fsdp, reward)
    reward_function/       # text2mask.py: cycle-consistency + mask-IoU reward
    format_prompt/         # prompt templates (non_thinking.jinja)
    datasets/              # scripts that build the RL parquet datasets
  transformers/            # VQ-SAM2 mask tokenizer + SAM2 model code
  vlm/                     # model + eval helpers (refcoco loaders, IoU metrics)
evaluation/
  gres/                    # referring segmentation (GRES)
  groundingsuite/          # GroundingSuite grounding
  gcg/                     # grounded caption generation (GCG)
  gar/                     # GAR-Bench VQA / detailed caption
  dlc_bench/               # dense-captioning eval (DLC-Bench)
  bbox/                    # bbox-format generalization variants

Setup

pip install -r requirements.txt
pip install -e .          # editable install of the verl package

Requires CUDA GPUs, PyTorch, and vLLM (SPMD mode). See requirements.txt.

Data & checkpoints

Configure the public training launcher with the following paths:

• MODEL_PATH — the cold-start (co-SFT) Qwen3-VL-4B + mask-token checkpoint that RL starts from. The public initialization is zhouyik/Qwen3-VL-4B-SAMTok-co.
• DATA_DIR — directory holding the RL *.parquet files. Build them with the scripts in projects/rl/datasets/ (e.g. prepare_dw_rl_dataset.py, prepare_gres_no_target_rl_dataset.py). A training mix typically combines dense-region (denseworld) + no-target (gres) parquets.
• <PATH_TO_COCO2014> — COCO2014 train2014/ images, used only by the gres/groundingsuite eval scripts.
• <PATH_TO_GAR_BENCH> — GAR-Bench annotations directory (holds GAR-Bench-VQA.json / GAR-Bench-Caption-Detailed.json and the images/), used by the GAR eval scripts. GAR-Bench is a separate public benchmark — get it from the official Grasp-Any-Region release.

Training

Download the public co-SFT initialization, prepare the three parquet files described in TRAIN.md, and launch:

hf download zhouyik/Qwen3-VL-4B-SAMTok-co \
  --local-dir checkpoints/Qwen3-VL-4B-SAMTok-co

MODEL_PATH=checkpoints/Qwen3-VL-4B-SAMTok-co \
DATA_DIR=data/cyclegrpo \
bash projects/rl/qwen3vl_4b_mt.sh

Defaults assume 1 node × 8 GPUs. The launcher reads paths from environment variables and checks all required parquet and SAMTok weight files before starting. WANDB_API_KEY is optional and WANDB_MODE defaults to offline.

Memory tuning (multi-image / OOM)

Multi-image samples produce long prompts; the actor backward can OOM. The main script has a commented block of levers — append them to the python3 -m verl.trainer.main command as needed:

• data.max_prompt_length / worker.rollout.max_num_batched_tokens — raise to fit long prompts (costs memory).
• worker.actor.micro_batch_size_per_device_for_{experience,update}=1 — smallest micro-batch.
• data.mini_rollout_batch_size=16 — smaller vLLM generation batch (lowers rollout-phase memory).
• worker.rollout.n should be a multiple of world_size (nnodes × n_gpus) when mixing cycle (region) and no-target sources, so the per-source sub-batches divide evenly across ranks.

Inference / evaluation

Each benchmark lives under evaluation/<benchmark>/. The multi-GPU launchers shard the dataset across GPUs and auto-merge:

# Referring segmentation (GRES)
bash evaluation/gres/run_gres_multigpu.sh                     8 <MODEL_PATH> ./results/gres/
# GroundingSuite
bash evaluation/groundingsuite/run_groundingsuite_multigpu.sh 8 <MODEL_PATH> ./results/groundingsuite/
# Grounded caption generation (GCG)
bash evaluation/gcg/run_gcg_multigpu.sh                       8 <MODEL_PATH> ./results/gcg/

# GAR-Bench VQA (single-process inference, then metrics)
python evaluation/gar/qwen3vl_gar_vqa_infer.py <MODEL_PATH> --output results/gar/vqa.json
python evaluation/gar/gar_vqa_metrics.py results/gar/vqa.json

# DLC-Bench (start the Llama judge server in a separate shell, then infer + eval)
bash evaluation/dlc_bench/serve_judge.sh
bash evaluation/dlc_bench/evaluate_dlc.sh <MODEL_PATH> <CACHE_NAME>
python evaluation/dlc_bench/eval_llama_without_image.py \
  --pred evaluation/dlc_bench/model_outputs/<CACHE_NAME>.json --base-url http://localhost:8007/v1

Fill the dataset placeholders inside the eval scripts where noted: <PATH_TO_COCO2014> (gres / groundingsuite) and <PATH_TO_GAR_BENCH> (gar). bbox-format generalization variants live in evaluation/bbox/.

Results

Base SAMTok (Qwen3-VL-4B) vs CycleGRPO (this work). Two CycleGRPO rows are reported: paper = the numbers in the ECCV 2026 paper, and release = the public checkpoint XinNUS/CycleGRPO-4B, a re-run that varies slightly from the paper (overall on par / marginally higher).

Region captioning — DLC-Bench (100 samples):

Text-to-mask — GroundingSuite (gIoU, %):

Region VQA — GAR-Bench-VQA (%):

Interleaved text-mask — GCG (METEOR / CIDEr / AP50 / mIoU / Recall):

Referring segmentation + target rejection — GRES (gIoU / cIoU / N-acc, %):

Citation

@inproceedings{cyclegrpo2026,
  title     = {Actor as Its Own Critic: Unifying Region Understanding and
               Localization via CycleGRPO},
  author    = {Zhang, Xin and Wang, Haochen and Zhou, Yikang and Wang, Zhuochen
               and Li, Xiangtai and Tan, Robby T.},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}

Acknowledgements

Built on EasyR1 and veRL; segmentation via SAM2. See LICENSE.