Dataset Card for Re-ranker Evaluation Datasets

This repo contains the evaluation datasets used in the paper "Language Model Re-rankers are Fooled by Lexical Similarities" accepted to FEVER 2025.

Dataset Details

The datasets in this repo are based on the NQ, LitQA2 (from LAB-Bench) and DRUID datasets. More details on the datasets can be found in our paper.

Uses

• Evaluate re-rankers.

Dataset Structure

We release the NQ, LitQA2 and DRUID datasets used in our re-ranker analysis. For LitQA2 we have two variants (LitQA2-o and LitQA2); one with the answer options appended and one without. We primarily use the former variation in our analysis as this corresponded to an improved re-ranker performance.

The datasets correspond to 3 different splits each:

• standard: The standard version of the dataset. For LitQA2, we only have the standard split since only LitQA2-o was used for the continued analysis.
• title: Same as the standard version, but with titles prepended to the passages.
• context: Same as the standard version, but with contextualizing contexts (generated by GPT-4o) prepended to the passages. This split does not exist for DRUID.
• prompt: Same as the standard version, but for which the queries have been formatted to better suit the re-ranking task. This split only exists for DRUID.

All datasets contains the following columns:

• nq_example_id/id/claim_id: The id column for NQ/LitQA2/DRUID.
• question: The query for which we would like to re-rank passages.
• chunks: The passages available given a corresponding query.
• gold_chunk_ix: The index of the gold chunk (in the list of chunks).

The datasets also contain various data-specific columns indicating other dataset-specific variables.

Dataset Creation

The dataset creation is described in our paper and in our GitHub repo.

Citation

@misc{languagemodelrerankersfooled,
      title={Language Model Re-rankers are Fooled by Lexical Similarities}, 
      author={Lovisa Hagström and Ercong Nie and Ruben Halifa and Helmut Schmid and Richard Johansson and Alexander Junge},
      year={2025},
      eprint={2502.17036},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.17036}, 
}