PangeaInstruct

Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages

🇪🇹 🇸🇦 🇧🇬 🇧🇩 🇨🇿 🇩🇪 🇬🇷 🇬🇧 🇺🇸 🇪🇸 🇮🇷 🇫🇷 🇮🇪 🇮🇳 🇮🇩 🇳🇬 🇮🇹 🇮🇱 🇯🇵 🇮🇩 🇰🇷 🇳🇱 🇲🇳 🇲🇾 🇳🇴 🇵🇱 🇵🇹 🇧🇷 🇷🇴 🇷🇺 🇱🇰 🇮🇩 🇰🇪 🇹🇿 🇱🇰 🇮🇳 🇮🇳 🇹🇭 🇹🇷 🇺🇦 🇵🇰 🇮🇳 🇻🇳 🇨🇳 🇹🇼

🏠 Homepage | 🤖 Pangea-7B | 📊 PangeaIns | 🧪 PangeaBench | 💻 Github | 📄 Arxiv | 📕 PDF | 🖥️ Demo

This README provides comprehensive details on the PangeaIns dataset, which was utilized during the instruction tuning phase for Pangea-7B.

Description of PangeaIns

PangeaIns is a 6M multilingual multicultural multimodal instruction tuning dataset spanning 39 languages.

PangeaIns Data Source

PangeaIns data path: PangeaIns.json (# samples: 6450624) PangeaIns data source:

Applications

PangeaIns was designed specifically for training the Pangea-7B model.

Code Instructions

The dataset follows the LLaVA data format. To retrieve all files from PangeaIns, use the following script:

from huggingface_hub import HfApi, hf_hub_download
import json

# Initialize the API client
api = HfApi()
dataset_name = "neulab/PangeaInstruct"

# Retrieve and download all files in the dataset
files = api.list_repo_files(repo_id=dataset_name, repo_type="dataset")

for file in files:
    hf_hub_download(repo_id=dataset_name, filename=file, repo_type="dataset")
    print(f"File downloaded: {file}")

# Load the complete PangeaIns dataset
with open('PangeaIns.json') as f:
  data = json.load(f)

Please note that image data is provided in compressed formats such as .tar or .zip. After downloading, you may need to extract these files to access the images. For images.tar files, you could untar them by running

tar -xvf images.tar

For images.zip files, you could unzip them by running

unzip images.zip

For some large tar files, we uploaded tar files splitted using the split command, such as split -n 4 -d images.tar part_. For example, in the cultural/laion-multi-1M subset, we splitted the images.tar file into 4 parts, part_00, part_01, part_02, and part_03. In such cases, you would need to first combine the splits and then extract the tar file.

cat part_* > images.tar
tar -xvf images.tar

Each subset within the PangeaIns dataset (e.g., ChartQA) contains a .json file for metadata and a corresponding .tar/.zip file for the images.

Citing the Dataset

BibTeX Citation:

@article{yue2024pangeafullyopenmultilingual,
  title={Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages},
  author={Xiang Yue and Yueqi Song and Akari Asai and Seungone Kim and Jean de Dieu Nyandwi and Simran Khanuja and Anjali Kantharuban and Lintang Sutawika and Sathyanarayanan Ramamoorthy and Graham Neubig},
  year={2024},
  journal={arXiv preprint arXiv:2410.16153},
  url={https://arxiv.org/abs/2410.16153}
}

Contact

Corresponding to: {xyue2,yueqis,gneubig}@cs.cmu.edu