We release the raw passages, embeddings, and index of MassiveDS.

Website: https://retrievalscaling.github.io

Versions

We release two versions of MassiveDS:

• MassiveDS-1.4T
  • MassiveDS-1.4T, which contains the embeddings and passages of the 1.4T-token datastore.
  • MasssiveDS-1.4T-raw-data, which contains the raw text of the 1.4T-token datastore.
• MassiveDS-140B
  • MassiveDS-140B, which contains the index, embeddings, passages, and raw text of a subsampled version containing 140B tokens in the datastore. We recommend downloading the raw data only and rebuild the index.

Note:

• We have uploaded all passages and embeddings for MassiveDS-1.4T. Please refer to the Github repo for detailed usage. You can rebuild the flat index directly using the embedding without doing extra computation. Due to file size limit by HF, we are looking for an althernative place to serve the indices. For direct access, please contact rulins@cs.washington.edu.
• MassiveDS-140B is ready to go. Please try our 10% subsampled version first!
• Code support to run with MassiveDS is in https://github.com/RulinShao/retrieval-scaling.

File structure

• raw_data: plain data in JSONL files.
• passages: chunked raw passages with passage IDs. Each passage is chunked to have no more than 256 words.
• embeddings: embeddings of the passages encoded with Contriever-MSMACRO.
• index: flat index built with embeddings.

Download

We recommend using Git LFS to download large files. We provide an example script below.

First, clone the Git history only to start working with it.

git clone --filter=blob:none https://huggingface.co/datasets/rulins/MassiveDS-1.4T
cd MassiveDS-1.4T/

(Optionally) Specify the directory that you want to partially download, e.g., the embeddings only. Skip this step if you want to download everything.

git sparse-checkout init --cone
git sparse-checkout set embeddings

Finally, pull the data.

git lfs install
git lfs pull