Revisiting Query Variation Robustness of Transformer Models

To reproduce our main table using the model-caches we published, you can simply have a look at the Jupyter Notebook in notebooks/main_table.ipynb. Our experimental results can all be found in the results folder. If you would like to run our code yourself, we highly recommend using the provided devcontainer. We provide a .devcontainer configuration-file that works with VSCode. To use it, install the the Dev Containers extension (ms-vscode-remote.remote-containers) and VSCode will ask you if you would like to open the project in a dev. container (say yes). This should take care of installing all the dependencies needed to run our code.

Our evaluation code can be found in src. To evaluate on of the embedding models, run src/main.py with the following arguments:

usage: main.py [-h] [-b BATCHSIZE] [--gpu | --no-gpu] [--cache | --no-cache] [--cache-only | --no-cache-only] {trec2019,antique} model {cos,rank}

positional arguments:
  {trec2019,antique}
  model
  {cos,rank}

options:
  -h, --help            show this help message and exit
  -b BATCHSIZE, --batchsize BATCHSIZE
  --gpu, --no-gpu
  --cache, --no-cache
  --cache-only, --no-cache-only

Here, model is one of the following: SBERT, CharBERT, E5Mistral, UAELarge, or Ada002.

Before you can evaluate on ANTIQUE, you should copy the top1000_*.csv files from data/top1000 into new .datasets/ folder (path relative to the working directory). These files are created by retrieving the top 1000 passages of the respective datasets and restructuring the runfile to follow the same structure as the top 1000 file of TREC DL '19 Passage.

If you would like to run the experiments using cached versions of each embedding model, you can download the caches here and extract them to .cache/.

Manually installing dependencies

To manually install required dependencies, install PyTorch, and the requirements.txt (note that CharacterBERT-DR should be installed without dependencies).

Artifacts

• Caches & Query Variation Training Set:
• Results can be found results folder