Modeling Ranking Properties with In-Context Learning

Implementation of LLM-based re-rankers with modeled ICL examples(e.g., fairness/diversity/etc.) using single and sliding window mechanisms.

Installation

To use the GPT_ranker functions directly, install using pip

pip install dist/GPT_ranker-1.0-py3-none-any.whl

NOTE: This package is a complete pipeline with phase-one retriever and re-ranker. Examples need to be generated separately following use-cases using the ICE class found in "GPT_ranker/modelling/ice.py"

Configuration

A configuration.py file contains all the key parameters, dataset locations, index, and the re-ranker details. You need to change the configuration values as required.

Single Window Re-ranking:

python -m GPT_ranker.main --window_size 20 --stride 10 --depth 100 --mode sliding --dataset touchev2 --retr bm25 --pgroup stance 

Fairness using ICL Sliding Window:

python -m GPT_ranker.main --window_size 20 --stride 10 --depth 100 --mode iclsliding --dataset touchev2 --pgroup stance --retr bm25 --icl FAR-touchev2.csv --seed 42 --icl_rank True --icl_depth 100

Parameters:

• --llm: Available LLM re-rankers 'rank-zephyr', 'rank-vicuna', 'gpt-4o' and 'gpt-4o-mini' (default gpt-4o-mini)
• --retr bm25/colbert (Updated to ColBERTv2)
• --fairness: True uses a slightly modified prompt for fair retrieval (Not Used)
• --mode: single/sliding/iclsingle/iclsliding modes (Updated iclsliding)
• --dataset: Currently supported > touchev2, fair22.eval, dl19 and dl20
• --icl: ICL example file name
• --pgroup: Protected group identifier
• --icl_rank: True: stage-one(bm25/ColBERT) based ordering of example documents | False: Random ordering of example documents
• --icl_depth: By default, it is 20 and considers examples in a single window manner. With 100, it slides over the example as per the window_size and stride parameters.
• --prompt: By default, the RankGPT Prompt is selected. 'FAIR' and 'DIVERSE' values use fairness and diverse instructions for re-ranking.

Evaluate Fairness:

python -m GPT_ranker.faireval --res_loc output/bm25.gpt-4o-mini.touchev2.t100.R42-W20.S10-FAR-ICLR.res --dataset touchev2 --pgroup stance

Diversity using ICL Sliding Window:

python -m GPT_ranker.main --window_size 20 --stride 10 --depth 100 --mode iclsliding --dataset dl19 --retr bm25 --icl DIV-dl19.csv --seed 42 --icl_rank True

Evaluate Diversity:

python -m GPT_ranker.noveleval --res_loc output/bm25.gpt-4o-mini.dl19.t100.R27182-W20.S10-DIV-ICLR.res --depth 100 --dataset dl19 --distance jaccard --threshold 0.9

NOTE: We suggest using file naming conventions for: example-set, dataset, and results, following the above examples for the proper functioning of the model.