MSMARCO

This repo is a study of reraking approachs for the MSMARCO passage ranking task. We use the MS MARCO dataset and the IR Datasets loader to load the data.

This repo main use Python with PyTorch and HuggingFace Transformers to implement the models. We also use a litte bit of Java with pyserini interface to run BM25.

Table of Contents

• Table of Contents
• Theory
  • Retrivers
    • BM25
    • Faiss
  • Rerankers
    • MonoBERT
    • SentenceTransformerSimilarity
• Installation
• Data
• Usage
  • Load Datasets
  • Run Experiments
  • Compile results
• Results
• Acknowledgements

Theory

We focus multistage reranking approachs. This means that we use a first stage to get a set of candidates using a fast way and then we use a second stage to rerank the candidates.

Our objective in the first layer is to maximize the recall, making sure that we get all the relevant documents. In the second layer, we use a more complex model to rerank the candidates trying to increase the precision.

Retrivers

For the first layers, called retriver, we basicly use BM25 with differente implementations and Faiss.

BM25

Regarding BM25, we use 3 different implementations:

1. BM25: The original BM25 implementation from python rank_bm25
2. BM25Fast: A faster implementation of BM25 using numba and numpy. This implementation is faster than the original one but it is not as accurate.
3. BM25PySerini: A Java implementation of BM25 using pyserini. This implementation is the most accurate one and it is also the fastest one. We use pyserini to run the Java code and get the results.

A simple resume of BM25 logic is that it use words frequency, inverse document frequency and some normalizations to calculate a score for each document. It is a fast way - depending on the implementation - to get a set of candidates with very high reall - also depending on the implementation due to text preprocessing. You can see a breif comparison between implementations and the impact of preprocessing in the results notebook.

Faiss

Faiss is a librar that implements Facebook AI Similarity Search. It uses embeddings to calculate the similarity between documents with some clever jumps to make it even fast. It is a very fast way to get a set of candidates and like we discussed in results it have enen high MRR than our best BM25. Going further it is almost better than BM25 + Similarity.

Rerankers

For the second layer, called reranker, we implemented and test some different models:

MonoBERT

MonoBERT is a BERT-based model that uses the BERT embeddings to calculate the similarity between documents. The bad thing about this model is that it is very slow since it needs to compute the embeddings for each document+query at runtime.

SentenceTransformerSimilarity

Here is like Faiss. It precomputes the embeddings and uses them to calculate the similarity between documents and the query embedding. The difference is that it uses a different model to compute the embeddings. It is also very fast.

Tutorials

We provide some tutorials to run some specific models or implement some specific techniques that we learn during the project. You can find them in the tutorials folder.

Installation

We recommend using a virtual environment to avoid dependency conflicts and use python >= 3.10.

pip install -r requirements.txt

Check pytorch to install cuda if you want.

Data

Our data is shared internally. Place it in the data folder.

Usage

We encapsulate the experiments run inside experiments.py CLI. You can run available experiments using the following commands:

Load Datasets

First of all, you need to load the datasets. Like we said on the Data section, we provide the data internally. After you download it, you can load the datasets using the following command:

python experiments.py load_datasets

Run Experiments

We create the framework in such way that you only need to run each layer/stage once. This means that if you run BM25 with dataset X, and after you want to run BM25 with dataset X and a new layer like MonoBERT, the framework will use the BM25 results cached (it will be stored in the results/raw folder).

The CLI interface is like:

python experiments.py run_experiment [dataset] [preproc] [*models]

Where:

• dataset is the subset:

  • subset_msmarco_train_0.01_99: 1% of the queries and using relevant docs plus 99 other docs.

• preproc is the preprocessing function:

  • none
  • lower: lower case
  • full: lower case, remove stop words, remove punctuation and do stemming

• models are the models you want to run (separated with spaces but if you want to run multiples in a single layer and create branches use -):

  • BM25
  • BM25Fast
  • BM25PySerini
  • Faiss
  • SentenceTransformerSimilarity
  • MonoBatchBERT
  • MonoBERT

Here is an example to run BM25 + SenteceTransformerSimilarity:

python experiments.py subset_msmarco_train_0.01_99 none BM25 SentenceTransformerSimilarity

And here how to run 2 experiments at once BM25 + MonoBERT and BM25 + MonoBatchBERT:

python experiments.py subset_msmarco_train_0.01_99 none BM25 MonoBERT-MonoBatchBERT

Compile results

We provide a comand to compile results to some simple tables with some usefull metrics(out main results are placed in results section).

Use the following command to compile and append results:

python experiments.py compile_results

Results

You can find the compiled results in the results/clean folder. The results are main stored in csv files. You can also find a notebook with some tables and plots of results in the results.ipynb.

Here is our main results:

Acknowledgements

• MS MARCO for the dataset.

• IR Datasets for the dataset loader.

• Transformers for the models and tokenizers.

• Rankify for the implementation of almost all used llm models.

• Vespa