[ICML 2026] RefChess: Training-Free Contextual Search for Zero-Shot Referring Image Segmentation

Official implementation for "RefChess: Training-Free Contextual Search for Zero-Shot Referring Image Segmentation".

Shiyan Tong • Jinxia Zhang • Zhiyuan Wang • Hao Tian • Yingying Wang • Kanjian Zhang • Haikun Wei
Southeast University • University of Electronic Science and Technology of China
Training-free · Zero-shot RIS · MCTS contextual search

🧭 Method

RefChess is a training-free framework for zero-shot referring image segmentation. It first generates class-agnostic proposals with SAM, scores them with FG-CLIP/GEM vision-language cues and language-derived constraints, and then uses Monte Carlo Tree Search to select the proposal that remains stable under sampled distractor contexts.

Motivation. Independent proposal ranking can be distracted by visually similar regions; RefChess favors proposals that remain reliable under contextual perturbations.

Overview of RefChess. SAM proposals are scored with language, region-text, object-centric, and heatmap evidence, then selected with stability-aware MCTS.

Qualitative comparisons on standard referring segmentation benchmarks.

📈 Performance

🛠️ Environment Setup

1. Clone the repository:

git clone https://github.com/Tongshiyan/RefChess.git
cd RefChess

2. Create a conda environment:

conda create -n refchess python=3.10 -y
conda activate refchess

3. Install dependencies:

conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia
pip install -r requirements.txt
python -m spacy download en_core_web_lg

4. Install the bundled SAM package:

cd third_party/segment_anything
pip install -e .
cd ../..

FG-CLIP weights are loaded through Hugging Face model IDs such as qihoo360/fg-clip-base and qihoo360/fg-clip2-base, so transformers.from_pretrained(...) will download them automatically on first use.

📦 Datasets and Pre-trained Models

Download the SAM checkpoint:

mkdir -p checkpoints
wget -P checkpoints https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth

Prepare RefCOCO, RefCOCO+, and RefCOCOg following the standard REFER/Zero-shot-RIS layout. A typical directory structure is:

path/to/datasets/COCO/refer/
  refs/
  instances.json
  train2014/

Pass the dataset root with --refer-data-root path/to/datasets/COCO/refer. If you use a different layout, adjust dataset.refer.IMAGE_DIR or provide symlinks to match the expected COCO train2014 image directory.

🚀 Quick Demo

Run RefChess on a single image/text pair:

python -m model.mcts \
  --image coco.jpg \
  --text "the man wearing a blue coat" \
  --sam-checkpoint checkpoints/sam_vit_h_4b8939.pth \
  --sims 2048 \
  --rollout-depth 10 \
  --save-best out_mcts/best.png \
  --save-trace-json out_mcts/trace.json

🧪 Evaluation

Evaluate on a RefCOCO-style split:

python main.py \
  --dataset refcoco \
  --split val \
  --splitBy unc \
  --refer-data-root path/to/datasets/COCO/refer \
  --sam-checkpoint checkpoints/sam_vit_h_4b8939.pth \
  --sims 2048 \
  --rollout-depth 10 \
  --object-neighbor-dist 0.3 \
  --max-samples 200 \
  --metrics-file out_mcts/refcoco_val.json

Useful options:

• --heatmap-backend: choose gem, fgclip, or hybrid.
• --fgclip-model-name: Hugging Face model ID or local checkpoint path for dense FG-CLIP heatmaps.
• --model-name: Hugging Face model ID or local checkpoint path for the FG-CLIP proposal-scoring backbone.
• --save-bad-cases: save low-IoU examples for inspection.

📝 Citation

If you find this work useful for your research, please cite our paper:

@inproceedings{tong2026refchess,
  title     = {RefChess: Training-Free Contextual Search for Zero-Shot Referring Image Segmentation},
  author    = {Tong, Shiyan and Zhang, Jinxia and Wang, Zhiyuan and Tian, Hao and Wang, Yingying and Zhang, Kanjian and Wei, Haikun},
  booktitle = {Proceedings of the 43rd International Conference on Machine Learning},
  year      = {2026},
  url       = {https://openreview.net/forum?id=RESRoG7nvd}
}

🙏 Acknowledgement

This project builds on excellent open-source work including Segment Anything, FG-CLIP, HybridGL, GEM, and REFER/Zero-shot-RIS tooling. We thank the authors for making their code and models available.