Evaluate your reranker quickly!

Usage

Install Pyserini and ColBERT, please refer to the original installation guide.

Run the following command to evaluate your reranker:

python eval_reranker.py --dataset $dataset --reranker $model

Supported datasets are listed in the file. Now, the eval code only supports BM25 as the first-stage retriever and only supports ColBERT and cross-encoder as the reranker.

Results

Coming soon!

Acknowledgement

• RankGPT: The code is mainly modified from this repository.
• Pyserini: Thanks for providing the pre-build indexes of MSMARCO and BEIR!