ZeroNLG

Without any labeled downstream pairs for training, ZeroNLG is an unified framework that deals with multiple natural language generation (NLG) tasks in a zero-shot manner, including image-to-text, video-to-text, and text-to-text generation tasks across English, Chinese, German, and French.

Pre-trained data: a machine-translated version of CC3M, including

• 1.1M English sentences
• 1.1M English-Chinese pairs
• 1.1M English-German pairs
• 1.1M English-French pairs

Paper: ZeroNLG: Aligning and Autoencoding Domains for Zero-Shot Multimodal and Multilingual Natural Language Generation

Authors: Bang Yang*, Fenglin Liu*, Yuexian Zou, Xian Wu, Yaowei Wang, David A. Clifton

Quick Start

Please follow our github repo to prepare the environment at first.

from zeronlg import ZeroNLG

# Automatically download the model from Huggingface Hub
# Note: this model is especially pre-trained for visual captioning
model = ZeroNLG('zeronlg-4langs-vc')

# `images` can be a remote image url, a local image/video file, etc
# `lang` should be one of English ('en'), Chinese ('zh'), German ('de'), and French ('fr')
url = 'https://img2.baidu.com/it/u=1856500011,1563285204&fm=253&fmt=auto&app=138&f=JPEG?w=667&h=500'
caption = model.forward(images=url, lang='en', num_beams=3, task='caption') 
# caption = "dogs play in the snow"

caption = model.forward(images=url, lang='zh', num_beams=3, task='caption') 
# caption = "狗 在 雪 地 里 玩 耍"

# Althernatively, you can call the specific forward function
caption = model.forward_caption(images=url, lang='en', num_beams=3)

Zero-Shot Performance

Visual captioning

Model: zeronlg-4langs-vc's multilingual decoder + CLIP's ViT-B-32 image encoder.

Notes:

• For non-English visual captioning, we do not report METEOR and SPICE, beacause they consider synonym matching and named entity recognition in English by default.
• For video captioning in English, we do not report SPICE following common practices.
• Flickr30K-Chinese is known as Flickr30K-CN.
• Flickr30K-German and Flickr30K-French are introduced in Multi30K.

Cross-modal retrieval

Model: zeronlg-4langs-vc's multilingual encoder + CLIP's ViT-B-32 image encoder

Notes:

• I2T: image-to-text retrieval, image as the query, search similar texts
• T2I: text-to-image retrieval, text as the query, search similar images
• R@K: Recall rate at top-K candidates
• Avg.: Average of R@{1,5,10} on both directions
• Retrieval uses the same testing sets as those for visual captioning, except COCO-1K, which splits the original testing set into 5 folds and report performance averaged over 5 folds.

Citation

@article{Yang2023ZeroNLG,
   title={ZeroNLG: Aligning and Autoencoding Domains for Zero-Shot Multimodal and Multilingual Natural Language Generation},
   author={Yang, Bang and Liu, Fenglin and Zou, Yuexian and Wu, Xian and Wang, Yaowei and Clifton, David A.},
   journal={arXiv preprint arXiv:2303.06458}
   year={2023}
}