We release the raw passages, embeddings, and index of MassiveDS.

Website: https://retrievalscaling.github.io

We release two versions of MassiveDS:

1. MassiveDS-1.4T, which contains 1.4T tokens in the datastore.
2. MassiveDS-140B, which is a subsampled version containing 140B tokens in the datastore.

File structure:

• raw_data: plain data in JSONL files.
• passages: chunked raw passages with passage IDs. Each passage is chunked to have no more than 256 words.
• embeddings: embeddings of the passages encoded with Contriever-MSMACRO.
• index: flat index built with embeddings.

Note: due to large data volume, we are still uploading the data for MassiveDS-1.4T. MassiveDS-140B is ready to go. Code support to run with MassiveDS is in https://github.com/RulinShao/retrieval-scaling.