rerankers-and-lexical-similarities

This is the repo for the paper "Language Model Re-rankers are Fooled by Lexical Similarities" accepted to FEVER 2025.

The paper is based on two main contributions:

1. the collection of samples from three diverse datasets that test the performance of re-rankers, and
2. careful evaluation of re-ranker performance based on novel metrics that compare re-ranker performance to a BM25 baseline.

The code underlying these contributions is described below.

Datasets for re-ranker evaluation

We collect samples from NQ, LitQA2 (from LAB-Bench) and DRUID for the evaluation of re-rankers. The samples are collected using the notebook src/gather_data.ipynb.

We have also uploaded the processed evaluation datasets to Hugging Face datasets, so one doesn't have to run the notebook mentioned above to get the evaluation datasets.

Evaluation of re-rankers

We first collect our metric of interest for the dataset samples using src/collect_metrics.ipynb. Our metrics of interest are:

• Various similarity scores (BERT scores, Jaccard similarities, BM25 scores)
• Various re-ranker scores (Cohere re-ranker, BGE re-ranker, and Jina re-rankers)

We also collect re-ranker scores from GPT-4o m and GPT-4o. This approach is described in Appendix E in our paper.

Finally, the results are plotted in src/evaluate.ipynb. For the analysis in the notebook, we assume that the data files with all relevant metrics have been prepared as described above.

Citation

@misc{languagemodelrerankersfooled,
      title={Language Model Re-rankers are Fooled by Lexical Similarities}, 
      author={Lovisa Hagström and Ercong Nie and Ruben Halifa and Helmut Schmid and Richard Johansson and Alexander Junge},
      year={2025},
      eprint={2502.17036},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.17036}, 
}