A Cross-lingual CLIP Model for Retrieving Classical Japanese Literature Images
In this research, we are introducing a cross-lingual CLIP model for retrieving classical Japanese literature images from English queries. The model transfers Japanese cultural knowledge from clip-japanese-base to XLM-RoBERTa, enabling multilingual image retrieval for large-scale classical literature archives.
This project proposes a cross-lingual CLIP model for retrieving classical Japanese literature images from English queries. Instead of training a CLIP model from scratch, the model adopts a teacher–student learning framework to transfer Japanese cultural knowledge from an existing Japanese CLIP model.
The teacher model is clip-japanese-base, which consists of a BERT-based text encoder and an EVA02-B image encoder trained on Japanese image–text pairs. The student model uses XLM-RoBERTa-base as a multilingual text encoder capable of processing English queries.
During training, Japanese captions are encoded by the teacher text encoder, while their English translations are encoded by the student encoder. The student output is mean-pooled and projected from 768 dimensions to the 512-dimensional embedding space used by CLIP through a fully connected projection layer. The model is optimized using Mean Squared Error (MSE) loss to align the student embeddings with the teacher embeddings.
After training, the teacher text encoder is discarded. During inference, English queries are encoded by the trained XLM-RoBERTa encoder and projected into the same multimodal embedding space as the frozen clip-japanese-base image encoder. Image retrieval is then performed using cosine similarity between text and image embeddings.
This architecture enables English-language retrieval while preserving the Japanese cultural understanding learned by the original Japanese CLIP model, making it suitable for searching large-scale classical Japanese literature archives.
A Cross-lingual CLIP Model for Retrieving Classical Japanese Literature Images
In this research, we are introducing a cross-lingual CLIP model for retrieving classical Japanese literature images from English queries. The model transfers Japanese cultural knowledge from clip-japanese-base to XLM-RoBERTa, enabling multilingual image retrieval for large-scale classical literature archives.
This project proposes a cross-lingual CLIP model for retrieving classical Japanese literature images from English queries. Instead of training a CLIP model from scratch, the model adopts a teacher–student learning framework to transfer Japanese cultural knowledge from an existing Japanese CLIP model.
The teacher model is clip-japanese-base, which consists of a BERT-based text encoder and an EVA02-B image encoder trained on Japanese image–text pairs. The student model uses XLM-RoBERTa-base as a multilingual text encoder capable of processing English queries.
During training, Japanese captions are encoded by the teacher text encoder, while their English translations are encoded by the student encoder. The student output is mean-pooled and projected from 768 dimensions to the 512-dimensional embedding space used by CLIP through a fully connected projection layer. The model is optimized using Mean Squared Error (MSE) loss to align the student embeddings with the teacher embeddings.
After training, the teacher text encoder is discarded. During inference, English queries are encoded by the trained XLM-RoBERTa encoder and projected into the same multimodal embedding space as the frozen clip-japanese-base image encoder. Image retrieval is then performed using cosine similarity between text and image embeddings.
This architecture enables English-language retrieval while preserving the Japanese cultural understanding learned by the original Japanese CLIP model, making it suitable for searching large-scale classical Japanese literature archives.
