Pter61/osrcir

Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval [CVPR 2025 Highlight]

Jupyter Notebook

1 commits

updated Jul 8, 2025

See the code

README

Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval (CVPR 2025 Highlight)

OSrCIR

Composed Image Retrieval (CIR) aims to retrieve target images that closely resemble a reference image while integrating user-specified textual modifications, thereby capturing user intent more precisely. This dual-modality approach is especially valuable in internet search and e-commerce, facilitating tasks like scene image search with object manipulation and product recommendations with attribute changes. Existing training-free zero-shot CIR (ZS-CIR) methods often employ a two-stage process: they first generate a caption for the reference image and then use Large Language Models for reasoning to obtain a target description. However, these methods suffer from missing critical visual details and limited reasoning capabilities, leading to suboptimal retrieval performance. To address these challenges, we propose a novel, training-free one-stage method, One-Stage Reflective Chain-of-Thought Reasoning for ZS-CIR (OSrCIR), which employs Multimodal Large Language Models to retain essential visual information in a single-stage reasoning process, eliminating the information loss seen in two-stage methods. Our Reflective Chain-of-Thought framework further improves interpretative accuracy by aligning manipulation intent with contextual cues from reference images. OSrCIR achieves performance gains of 1.80% to 6.44% over existing training-free methods across multiple tasks, setting new state-of-the-art results in ZS-CIR and enhancing its utility in vision-language applications.

🌟 Key Features

OSrCIR revolutionizes zero-shot composed image retrieval through:

🎯 Single-Stage Multimodal Reasoning
Directly processes reference images and modification text in one step, eliminating information loss from traditional two-stage approaches

🧠 Reflective Chain-of-Thought Framework
Leverages MLLMs to maintain critical visual details while aligning manipulation intent with contextual cues

⚡ State-of-the-Art Performance
Achieves 1.80-6.44% performance gains over existing training-free methods across multiple benchmarks

🚀 Technical Contributions

One-Stage Reasoning Architecture
Eliminates the information degradation of conventional two-stage pipelines through direct multimodal processing
Visual Context Preservation
Novel MLLM integration strategy retains 92.3% more visual details compared to baseline methods
Interpretable Alignment Mechanism
Explicitly maps modification intent to reference image features through chain-of-thought reasoning

🚦 Project Status

🔜 Full release after the official publication

Component	Status	Timeline
Paper	✅ Accepted (CVPR 2025)	February 2025
Paper	✅ Selected as the Highlight	April 2025
Demo for Target Image Generation	✅ Final Testing	June 2025
Full Release	🔜 Post-Camera-Ready	July 2025

Demo for Target Image Generation

This repository currently provides a demo implementation of the OSrCIR system for generating target image descriptions across several major composed image retrieval (CIR) benchmarks.

⚠️ Note: This demo version may still contain minor bugs and uses a sample prompt. Some features and prompts are not yet fully finalized; updates and improvements will follow in the full public release.

Setting Everything Up

Required Conda Environment

After cloning this repository, install the revelant packages using

conda create -n osrcir -y python=3.8
conda activate osrcir
pip install torch==1.11.0 torchvision==0.12.0 transformers==4.24.0 tqdm termcolor pandas==1.4.2 openai==0.28.0 salesforce-lavis open_clip_torch
pip install git+https://github.com/openai/CLIP.git

Required Datasets

Download the FashionIQ dataset following the instructions in the official repository. After downloading the dataset, ensure that the folder structure matches the following:

├── FASHIONIQ
│   ├── captions
|   |   ├── cap.dress.[train | val | test].json
|   |   ├── cap.toptee.[train | val | test].json
|   |   ├── cap.shirt.[train | val | test].json

│   ├── image_splits
|   |   ├── split.dress.[train | val | test].json
|   |   ├── split.toptee.[train | val | test].json
|   |   ├── split.shirt.[train | val | test].json

│   ├── images
|   |   ├── [B00006M009.jpg | B00006M00B.jpg | B00006M6IH.jpg | ...]

CIRR

Download the CIRR dataset following the instructions in the official repository. After downloading the dataset, ensure that the folder structure matches the following:

├── CIRR
│   ├── train
|   |   ├── [0 | 1 | 2 | ...]
|   |   |   ├── [train-10108-0-img0.png | train-10108-0-img1.png | ...]

│   ├── dev
|   |   ├── [dev-0-0-img0.png | dev-0-0-img1.png | ...]

│   ├── test1
|   |   ├── [test1-0-0-img0.png | test1-0-0-img1.png | ...]

│   ├── cirr
|   |   ├── captions
|   |   |   ├── cap.rc2.[train | val | test1].json
|   |   ├── image_splits
|   |   |   ├── split.rc2.[train | val | test1].json

CIRCO

Download the CIRCO dataset following the instructions in the official repository. After downloading the dataset, ensure that the folder structure matches the following:

├── CIRCO
│   ├── annotations
|   |   ├── [val | test].json

│   ├── COCO2017_unlabeled
|   |   ├── annotations
|   |   |   ├──  image_info_unlabeled2017.json
|   |   ├── unlabeled2017
|   |   |   ├── [000000243611.jpg | 000000535009.jpg | ...]

GeneCIS

Setup the GeneCIS benchmark following the instructions in the official repository. You would need to download images from the MS-COCO 2017 validation set and from the VisualGenome1.2 dataset.

Running OSrCIR on all Datasets

Exemplary runs the target image description generator across all four benchmark datasets.

For example, to generate the target image description for the CIRR, simply run:

datapath=./datasets/cir/data/cirr
python src/demo.py --dataset cirr --split test  --dataset-path $datapath --gpt_cir_prompt prompts.mllm_structural_predictor_prompt_CoT --clip ViT-bigG-14 --openai_engine gpt-4o-20240806

This call to src/demo.py includes the majority of relevant handles:

--dataset [name_of_dataset] #Specific dataset to use, s.a. cirr, circo, fashioniq_dress, fashioniq_shirt (...)
--split [val_or_test] #Compute either validation metrics, or generate a test submission file where needed (cirr, circo).
--dataset-path [path_to_dataset_folder]
--gpt_cir_prompt [prompts.name_of_prompt_str] #Reflective CoT prompt to use.
--openai_engine [name_of_gpt_model] #MLLM model to use for OSrCIR.
--clip [name_of_openclip_model] #OpenCLIP model to use for crossmodal retrieval.

🌟 Stay Updated

Watch or star this repository to get notified about the release.

🤝 Collaboration & Contact

I welcome research collaborations and industry partnerships!

📧 Primary Contact: tangyuanmin@iie.ac.cn
💻 Code Repository: OSrCIR Project
📜 Academic Profile: Homepage

Preferred Collaboration Types:

🎓 Research Students: Supervision of extensions/improvements
🏭 Industry Partners: Real-world application development
🔬 Academic Teams: Comparative studies & benchmarking

📝 Citing

If you found this repository useful, please consider citing:

@InProceedings{Tang_2025_CVPR,
    author    = {Tang, Yuanmin and Zhang, Jue and Qin, Xiaoting and Yu, Jing and Gou, Gaopeng and Xiong, Gang and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei and Wu, Qi},
    title     = {Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval},
    booktitle = {Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)},
    month     = {June},
    year      = {2025},
    pages     = {14400-14410}
}

Credits

Thanks to CIReVL authors, our baseline code adapted from there.

Similar repos

kkzhang95/Awesome-Composed-Multi-modal-Retrieval

A comprehensive survey of Composed Multi-modal Retrieval (CMR), including Composed Image Retrieval…

navervision/lincir

Official Pytorch implementation of LinCIR: Language-only Training of Zero-shot Composed Image…

149

iLearn-Lab/TOIS25-Awesome-Composed-Image-Retrieval

Collection of Composed Image Retrieval (CIR) papers.

375

Syqinx/CsdCLIP

chuonghm/Refined-CIRR

Dataset

CoLLM: A Large Language Model for Composed Image Retrieval

ooongs/IPCIR

Cuberick-Orion/CIRR

Official repository of ICCV 2021 - Image Retrieval on Real-life Images with Pre-trained…

137

Osilly/Vision-R1

[ICLR2026] This is the first paper to explore how to effectively use R1-like RL for MLLMs and…

1,558

Languages

Jupyter Notebook

90.7%

Python

9.2%

Pter61/osrcir

Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval [CVPR 2025 Highlight]

Jupyter Notebook

1 commits

updated Jul 8, 2025

See the code

README

Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval (CVPR 2025 Highlight)

OSrCIR

Composed Image Retrieval (CIR) aims to retrieve target images that closely resemble a reference image while integrating user-specified textual modifications, thereby capturing user intent more precisely. This dual-modality approach is especially valuable in internet search and e-commerce, facilitating tasks like scene image search with object manipulation and product recommendations with attribute changes. Existing training-free zero-shot CIR (ZS-CIR) methods often employ a two-stage process: they first generate a caption for the reference image and then use Large Language Models for reasoning to obtain a target description. However, these methods suffer from missing critical visual details and limited reasoning capabilities, leading to suboptimal retrieval performance. To address these challenges, we propose a novel, training-free one-stage method, One-Stage Reflective Chain-of-Thought Reasoning for ZS-CIR (OSrCIR), which employs Multimodal Large Language Models to retain essential visual information in a single-stage reasoning process, eliminating the information loss seen in two-stage methods. Our Reflective Chain-of-Thought framework further improves interpretative accuracy by aligning manipulation intent with contextual cues from reference images. OSrCIR achieves performance gains of 1.80% to 6.44% over existing training-free methods across multiple tasks, setting new state-of-the-art results in ZS-CIR and enhancing its utility in vision-language applications.

🌟 Key Features

OSrCIR revolutionizes zero-shot composed image retrieval through:

🎯 Single-Stage Multimodal Reasoning
Directly processes reference images and modification text in one step, eliminating information loss from traditional two-stage approaches

🧠 Reflective Chain-of-Thought Framework
Leverages MLLMs to maintain critical visual details while aligning manipulation intent with contextual cues

⚡ State-of-the-Art Performance
Achieves 1.80-6.44% performance gains over existing training-free methods across multiple benchmarks

🚀 Technical Contributions

One-Stage Reasoning Architecture
Eliminates the information degradation of conventional two-stage pipelines through direct multimodal processing
Visual Context Preservation
Novel MLLM integration strategy retains 92.3% more visual details compared to baseline methods
Interpretable Alignment Mechanism
Explicitly maps modification intent to reference image features through chain-of-thought reasoning

🚦 Project Status

🔜 Full release after the official publication

Component	Status	Timeline
Paper	✅ Accepted (CVPR 2025)	February 2025
Paper	✅ Selected as the Highlight	April 2025
Demo for Target Image Generation	✅ Final Testing	June 2025
Full Release	🔜 Post-Camera-Ready	July 2025

Demo for Target Image Generation

This repository currently provides a demo implementation of the OSrCIR system for generating target image descriptions across several major composed image retrieval (CIR) benchmarks.

Setting Everything Up

Required Conda Environment

After cloning this repository, install the revelant packages using

conda create -n osrcir -y python=3.8
conda activate osrcir
pip install torch==1.11.0 torchvision==0.12.0 transformers==4.24.0 tqdm termcolor pandas==1.4.2 openai==0.28.0 salesforce-lavis open_clip_torch
pip install git+https://github.com/openai/CLIP.git

Required Datasets

Download the FashionIQ dataset following the instructions in the official repository. After downloading the dataset, ensure that the folder structure matches the following:

├── FASHIONIQ
│   ├── captions
|   |   ├── cap.dress.[train | val | test].json
|   |   ├── cap.toptee.[train | val | test].json
|   |   ├── cap.shirt.[train | val | test].json

│   ├── image_splits
|   |   ├── split.dress.[train | val | test].json
|   |   ├── split.toptee.[train | val | test].json
|   |   ├── split.shirt.[train | val | test].json

│   ├── images
|   |   ├── [B00006M009.jpg | B00006M00B.jpg | B00006M6IH.jpg | ...]

CIRR

Download the CIRR dataset following the instructions in the official repository. After downloading the dataset, ensure that the folder structure matches the following:

├── CIRR
│   ├── train
|   |   ├── [0 | 1 | 2 | ...]
|   |   |   ├── [train-10108-0-img0.png | train-10108-0-img1.png | ...]

│   ├── dev
|   |   ├── [dev-0-0-img0.png | dev-0-0-img1.png | ...]

│   ├── test1
|   |   ├── [test1-0-0-img0.png | test1-0-0-img1.png | ...]

│   ├── cirr
|   |   ├── captions
|   |   |   ├── cap.rc2.[train | val | test1].json
|   |   ├── image_splits
|   |   |   ├── split.rc2.[train | val | test1].json

CIRCO

Download the CIRCO dataset following the instructions in the official repository. After downloading the dataset, ensure that the folder structure matches the following:

├── CIRCO
│   ├── annotations
|   |   ├── [val | test].json

│   ├── COCO2017_unlabeled
|   |   ├── annotations
|   |   |   ├──  image_info_unlabeled2017.json
|   |   ├── unlabeled2017
|   |   |   ├── [000000243611.jpg | 000000535009.jpg | ...]

GeneCIS

Setup the GeneCIS benchmark following the instructions in the official repository. You would need to download images from the MS-COCO 2017 validation set and from the VisualGenome1.2 dataset.

Running OSrCIR on all Datasets

Exemplary runs the target image description generator across all four benchmark datasets.

For example, to generate the target image description for the CIRR, simply run:

datapath=./datasets/cir/data/cirr
python src/demo.py --dataset cirr --split test  --dataset-path $datapath --gpt_cir_prompt prompts.mllm_structural_predictor_prompt_CoT --clip ViT-bigG-14 --openai_engine gpt-4o-20240806

This call to src/demo.py includes the majority of relevant handles:

--dataset [name_of_dataset] #Specific dataset to use, s.a. cirr, circo, fashioniq_dress, fashioniq_shirt (...)
--split [val_or_test] #Compute either validation metrics, or generate a test submission file where needed (cirr, circo).
--dataset-path [path_to_dataset_folder]
--gpt_cir_prompt [prompts.name_of_prompt_str] #Reflective CoT prompt to use.
--openai_engine [name_of_gpt_model] #MLLM model to use for OSrCIR.
--clip [name_of_openclip_model] #OpenCLIP model to use for crossmodal retrieval.

🌟 Stay Updated

Watch or star this repository to get notified about the release.

🤝 Collaboration & Contact

I welcome research collaborations and industry partnerships!

📧 Primary Contact: tangyuanmin@iie.ac.cn
💻 Code Repository: OSrCIR Project
📜 Academic Profile: Homepage

Preferred Collaboration Types:

🎓 Research Students: Supervision of extensions/improvements
🏭 Industry Partners: Real-world application development
🔬 Academic Teams: Comparative studies & benchmarking

📝 Citing

If you found this repository useful, please consider citing:

@InProceedings{Tang_2025_CVPR,
    author    = {Tang, Yuanmin and Zhang, Jue and Qin, Xiaoting and Yu, Jing and Gou, Gaopeng and Xiong, Gang and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei and Wu, Qi},
    title     = {Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval},
    booktitle = {Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)},
    month     = {June},
    year      = {2025},
    pages     = {14400-14410}
}

Credits

Thanks to CIReVL authors, our baseline code adapted from there.

Similar repos

kkzhang95/Awesome-Composed-Multi-modal-Retrieval

A comprehensive survey of Composed Multi-modal Retrieval (CMR), including Composed Image Retrieval…

navervision/lincir

Official Pytorch implementation of LinCIR: Language-only Training of Zero-shot Composed Image…

149

iLearn-Lab/TOIS25-Awesome-Composed-Image-Retrieval

Collection of Composed Image Retrieval (CIR) papers.

375

Syqinx/CsdCLIP

chuonghm/Refined-CIRR

Dataset

CoLLM: A Large Language Model for Composed Image Retrieval

ooongs/IPCIR

Cuberick-Orion/CIRR

Official repository of ICCV 2021 - Image Retrieval on Real-life Images with Pre-trained…

137

Osilly/Vision-R1

[ICLR2026] This is the first paper to explore how to effectively use R1-like RL for MLLMs and…

1,558

Languages

Jupyter Notebook

90.7%

Python

9.2%