🌍 PALO: A Polyglot Large Multimodal Model for 5B People

Vision-language conversation in English, Chinese, French, Spanish, Russian, Japanese, Arabic, Hindi, Bengali and Urdu

Multi-lingual Training Dataset

This repository contains the multilingual, multimodal dataset used to train PALO. The dataset includes 665K English instructions from LLaVA-v1.5 and translations of LLaVA-Instruct-150K into Chinese, French, Spanish, Russian, Japanese, Arabic, Hindi, Bengali, and Urdu, totaling nearly 2.1M instructions. Please refer to the Section # 3.1 of our paper for details.

Prepare image-data

Please download the images from constituting datasets,

• COCO: train2017
• GQA: images
• OCR-VQA: download script, save all files as .jpg
• TextVQA: train_val_images
• VisualGenome: part1, part2

After downloading all of them, organize the data as follows in PALO/data,

├── coco
│   └── train2017
├── gqa
│   └── images
├── ocr_vqa
│   └── images
├── textvqa
│   └── train_images
└── vg
    ├── VG_100K
    └── VG_100K_2