Composed Multi-modal Retrieval: A Survey of Approaches and Applications

This repo is used for recording and tracking recent Composed Multi-modal Retrieval (CMR) works, including Composed Image Retrieval (CIR), Composed Video Retrieval (CVR), Composed Person Retrieval (CPR), etc.

The survey can be found here.

If you find this survey helpful, please consider citing:

@article{Zhang2025Composed,
  title   = {Composed Multi-modal Retrieval: A Survey of Approaches and Applications},
  author  = {Zhang, Kun and Li, Jingyu and Li, Zhe and Zhang, Jingjing and others},
  journal = {arXiv preprint arXiv:2503.01334},
  year    = {2025}
}

This repository is still a work in progress. If you find any work missing or have any suggestions, feel free to pull requests. We will add the missing papers to this repo ASAP.

What is Composed-Multi-modal-Retrieval (CMR)?

Figure1 (a) The evolution of content-based retrieval technology. (b) In the current research on composed multimodal retrieval (CMR), three main paradigms have been developed. (c) The CMR applications, broadly categorized based on application scenarios and image domains.

Generally, the evolution of content-based retrieval technology has witnessed the transformation from Unimodal Retrieval (UR) to Cross-modal Retrieval (CR), and then to Composed Multi-modal Retrieval (CMR). Compared with early-stage unimodal retrieval, which was limited to querying information within the same modality, as shown in Fig.1(a1), cross-modal retrieval has achieved remarkable accuracy and widespread application in the present era. This enables the search for semantically relevant content in one modality based on the instance query from another modality, e.g., using text search on images in Fig.1(a2), allowing users to make full use of these heterogeneous data. In recent years, composed multi-modal retrieval has emerged as a thriving content-based retrieval technology. Within this technical framework, as depicted in Fig.1(a3), the system aims to discover images/videos that not only bear resemblance to the given reference image/video but also allow for specific modifications based on the provided textual feedback from the user. In this sense, CMR pioneers an advanced level of interactive and conditional retrieval mechanisms, leveraging deep integration of visual and linguistic information. This integration greatly enhances the flexibility and precision of user-expressed search intents, injecting new vitality into domains such as internet search and e-commerce. Consequently, CMR exhibits vast potential and far-reaching impact as the next-generation content-based retrieval engine in real-world application scenarios.

What are the challenges and existing lines of research in CMR?

A core of CMR is that it requires a synergistic understanding and composition of both input vision and language information as the multi-modal query. The earliest closely related studies of CMR are in the field of attribute-based fashion image retrieval, where the key difference is that the textual feedback in attribute-based fashion image retrieval is limited to the predefined attribute value (e.g., 'mini', 'white', 'red'), while CMR is the natural language with multiple words (e.g., 'showing this animal of the input image facing the camera under sunlight'), which is more flexible yet challenging. The pioneering CMR works introduced a framework where input queries combine an image with natural language instructions for desired modifications, sparking numerous subsequent approaches. Current research in CMR is primarily focused on three paradigms: (1) supervised learning-based CMR (SL-CMR), which focuses on how to design a better combination mechanism of vision and language through supervised training of annotated data, (2) zero-shot learning-based CMR (ZSL-CMR), which focuses on how to simulate and build a visual-linguistic multi-modal information combination framework without annotated data, and (3) semi-supervised learning-based CMR (SSL-CMR), which focuses on how to enhance the learning of visual-linguistic combination via generated pseudo-labeling data.

Supervised Learning-based CMR (SL-CMR)

For the SL-CMR pipeline, a notable characteristic is the requirement of annotated triplet data $(I_{r}, T_{m}, I_{t})$, which denotes the reference query image, the modified text, and the ground-truth target image, respectively. As illustrated in Fig.1(b1), for the given inputs $I_{r}$ and $T_{m}$, SL-CMR involves mining the content that should be modified in the reference image $I_{r}$ according to the text $T_{m}$, so as to learn a multi-modal compositional embedding to find the interested target image $I_{t}$. Thus, the challenges faced by SL-CMR mainly lies in addressing two issues: "Where to see", which refers to attending to the content in the reference image that needs change, and "How to change", which aims to modify the reference image based on the textual information while preserving the remaining information. In recent years, research on SL-CMR has primarily focused on three aspects: (1) data construction, focusing on labeling triples with accurate semantic difference descriptions;(2) model architecture, focusing on designing a better vision-language combiner via cross-modal feature alignment and fusion strategies, as well as the design of other novel frameworks that can be plugged; (3) loss optimization, focusing on the design of more reasonable feature combination constraints. Although supervised training relying on these carefully labeled data often offers high performance, SL-CMR inherently faces two shortcomings: 1) annotating such triplets is both difficult and labor-intensive, and 2) the supervised approaches trained on the collected limited and specific triplets are also hard for generalization.

1. Data Construction Approaches

2. Model Architecture Approaches

2.1 Design of the visual-linguistic combiner

2.2 Design of plug and play structure

2.3 Design of non-explicit combiner architecture

3. Loss Optimization Approaches

3.1 Based on contrastive learning

3.2 Consistency constraint

3.3 Multi-task joint loss

Zero-Shot Learning-based CMR (ZSL-CMR)

Recently, composed multi-modal retrieval based on zero-shot learning has been proposed to address the above limitations. During the training process, the model is trained solely on easily obtainable image-text pairs, without the need for annotated triplets. Its training process usually revolves around learning the modality transformers, which simulate the combination of visual and linguistic information in testing. As a result, the training and testing phases typically involve different network structures. Thus, the main challenge of ZSL-CMR lies in designing transformation frameworks that are equivalent to supervised learning in the absence of supervision signals, aiming to maximize the zero-shot generalization ability as much as possible. To address this challenge, the academic community has developed strategies across three key aspects: (1) Image-side transformation: this approach focuses on learning the implicit or explicit visual-to-linguistic transformation using images as input. During testing, it converts the reference image into a query that can be integrated with the relative textual information; (2) Text-side transformation: in this approach, text is used as input to simulate image features, constructing a training framework that relies solely on language. During testing, the model directly takes image inputs; (3) External knowledge assistance: this approach explores the utilization of additional knowledge to enhance details, such as attributes and colors, thereby improving retrieval performance.

1. Image-side Transformation Approaches

1.1 Explicit visual Transformation

1.2 Implicit visual Transformation

2. Text-side Transformation Approaches

3. External Knowledge Assistance Approaches

Semi-Supervised Learning-based CMR (SSL-CMR)

Although zero-shot combined multimodal retrieval does not rely on labeled data, its performance is often lower than supervised training, which brings obstacles to the application of the model. To alleviate this problem, some works have proposed a semi-supervised combined multimodal learning paradigm based on automatically generated triple data. In this setting, relying on the relatively easy-to-obtain image-text data, existing SSL-CMR work mainly generates triplet data from two aspects: (1) generating images, such as editing the input reference image according to the conditional text to create the target image; (2) generating text, such as describing the difference caption between the two input images. By generating these triplet data, the model can not only capture the combination of vision and language more accurately during learning, but also avoid the limitations of cumbersome annotation. Although the generated data may be noisy, it also combines the advantages of supervision and zero-shot learning, which is a promising direction.

1. Automatic Data Construction

1.1 Generating Language Content

1.2 Generating Language and Visual Content

2. Noise/Uncertainty in Data

2.1 Content Noise

2.2 Annotation Uncertainty

Evaluation Metrics

Application

Research in CMR has vast application potential. It can be broadly categorized based on application scenarios and image domain differences, including domains as fashion and e-commerce images, natural images, videos, remote sensing images, person images, sketch images, and interactive conversation. The specific application can be personalized product shopping, media search, event discovery, environmental monitoring, law enforcement, customer service bots, and so on. In summary, CMR represents a paradigm shift in search systems by integrating visual and textual modalities. These systems enable fine-grained, context-aware, and user-centric searches across diverse domains, offering significant improvements in both retrieval accuracy and user satisfaction.

(1) Composed Fashion Image Retrieval and Composed Natural Image Retrieval

Traditional fashion image retrieval primarily relies on simple image search or keyword-based search. However, these methods often fail to meet user needs when searching for specific fashion items with complex attributes such as color, style, and material. The main goal of Composed Fashion Image Retrieval (CFIR) is to achieve more accurate and personalized fashion searches by combining both images and texts. CFIR has broad applications in e-commerce. By combining image-based search with textual refinement, it improves search efficiency and helps users locate products that better match their preferences. For example, users can input a reference image (e.g., a photo of a clothing item) along with a textual modification (e.g., "long-sleeve version"). This leads to higher customer satisfaction and greater user engagement. For retailers and brands, CFIR supports personalized product recommendations, fashion trend analysis, and more effective inventory management.

(2) Composed Natural Image Retrieval

Traditional image retrieval systems typically rely on either visual or textual inputs alone. However, single-modality approaches are limited in their ability to represent complex queries involving multiple attributes or concepts. Composed Natural Image Retrieval (CNIR) addresses this limitation by integrating both textual descriptions and image content, enabling systems to better understand abstract and nuanced user requirements. For example, users can upload an image of a favorite landscape along with a description such as “same location but in Autumn,” and the CNIR system will analyze both modalities to retrieve semantically relevant images. By enabling more accurate and personalized search experiences, CNIR significantly improves efficiency and user satisfaction. It also opens new opportunities for innovative services and the broader development of advanced image retrieval technologies.

(3) Composed Video Retrieval

Composed Video Retrieval (CoVR) enables the retrieval of specific videos from large databases by integrating visual queries with textual modification instructions, allowing for more precise and context-aware searches. This approach overcomes the limitations of traditional content-based video retrieval systems, which rely solely on visual features and often fail to capture user intent or nuanced context. The primary objective of CoVR is to improve search accuracy by leveraging multi-modal inputs. CoVR holds strong application potential across multiple domains, including online video platforms, live event discovery, and sports video retrieval. On video platforms, it supports advanced content recommendation and management systems by identifying and suggesting videos that better align with user preferences and interests.

(4) Composed Remote Sensing Image Retrieval

Composed Remote Sensing Image Retrieval (CRSIR) enables users to perform more precise and expressive searches by combining both visual and textual inputs. Instead of relying on a single modality, users can submit a reference image together with a textual description that specifies desired geographic features, environmental conditions, or temporal information. This multi-modal approach enhances the system’s ability to interpret complex queries, leading to more accurate and context-aware retrieval results.

(5) Composed Person Retrieval

Composed Person Retrieval (CPR) represents an innovative approach to identifying specific individuals by leveraging both visual and textual information. Traditional methods, such as Image-based Person Retrieval (IPR) and Text-based Person Retrieval (TPR) , often fall short in effectively utilizing both types of data, leading to a loss in accuracy. CPR aims to address this limitation by simultaneously employing image and text queries to enhance the retrieval process. This dual-modality approach not only increases the descriptive power of the query but also refines the relevance of search results, providing more accurate identification of target individuals. CPR is particularly useful in social services and public security, where precise person identification is crucial.

(6) Composed Sketch-based Image Retrieval

Composed Sketch-Text Image Retrieval aims to improve the accuracy and relevance of image retrieval by integrating sketch-based and textual inputs. This approach leverages sketches to capture object shapes and structures, while textual descriptions provide complementary details such as color, material, and texture. By combining coarse structural information with fine-grained attributes, it enables more expressive and flexible querying, especially useful when users lack a specific reference image.

(7) Interactive/Conversational Retrieval

Interactive/Conversational Retrieval (ICR) represents an advanced approach to image retrieval that leverages natural language interactions between users and systems to refine search outcomes progressively. Unlike traditional methods relying solely on images or predefined textual attributes, ICR integrates user feedback through conversational interfaces, enhancing the accuracy and relevance of search results. This method enables users to provide iterative feedback in natural language, refining queries dynamically until they locate the desired image or item. The primary objective of ICR is to facilitate more intuitive, precise, and personalized searches by incorporating both visual and semantic information effectively. ICR has significant applications across various domains, including e-commerce, fashion, and social media.

Copyright Notice

All content in this repository, including but not limited to text, images, and code, is the intellectual property of Kun Zhang and is protected under applicable copyright laws. Since this repository contains content unpublished, any further reproduction, distribution, display, or performance of this content is strictly prohibited without prior written permission from Kun Zhang. (Jun 23, 2024)