CoRank

Table of Contents

• Overview
• Core Challenges
• Key Contributions
• Experimental Results
• Project Structure
• Installation
• Data & Results
• Reproduction
• License

Overview

This project presents a training-free, model-agnostic reranking framework tailored for scientific document retrieval. By leveraging compact semantic representations, it overcomes two key obstacles in LLM-based listwise reranking: (1) suboptimal first-stage retrieval that buries relevant papers deep in the candidate list, and (2) context-window limits that force full-text rerankers to consider only a handful of documents.

Core Challenges

1. Scientific retrieval often produces a noisy initial ranking, so many relevant documents never surface.
2. Standard listwise rerankers must load full documents into the LLM's context, drastically capping the number of candidates—and thus the final recall.

Key Contributions

• Compact document features (categories, sections, keywords) extracted offline.
• Three-stage hybrid reranking:
  1. Offline feature extraction.
  2. Coarse rerank on compact representations to expand coverage.
  3. Fine-grained rerank on the top candidates using full text.
• No additional model training; works with any LLM backend.

Experimental Results

On scientific literature benchmarks, our approach significantly improves nDCG@10 scores across multiple LLM architectures, demonstrating the power of semantic feature extraction for scientific reranking.

Project Structure

├── script/
│   ├── rerank/
│   │   ├── analyze/   # Dataset analysis scripts
│   │   └── baseline/  # Baseline reranking evaluation scripts
│
├── src/
│   ├── api/           # LLM API interfaces (online and batched processing)
│   ├── data/          # Data loading and preprocessing modules
│   ├── eval_rerank.py # Main reranking evaluation script
│   ├── metric/        # Evaluation metrics
│   ├── prerank/       # Initial ranking and document analysis
│   ├── ranker/        # Dense retrieval models
│   ├── reranker/      # LLM-based reranking implementations
│   └── util/          # Utility functions and configurations
│
├── data/              # Raw and processed datasets (gitignored)
├── res/               # Results and model outputs (gitignored)
└── README.md          # Project documentation

Note: The data/ and res/ directories are gitignored as they contain large datasets and experimental results.

Installation

Install dependencies (recommended to use a virtual environment):

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Data & Results

All datasets and output results can be downloaded from Google Drive:

• Raw & processed data → link
• Retrieval results & logs → link

Fetch and organize everything under data/ and res/.

A newer version of data and results that include further analysis and complete datasets can be found here

Reproduction

To reproduce the main experiments and complementary experiments, run the baseline reranking scripts:

# Basic reranking evaluation
python -m src.eval_rerank --dataset_name <dataset> --rerank_strategy <strategy> --temperature <temp> --random_seed <seed>

# Run baseline experiments
bash script/rerank/baseline/gemini_full_10.sh

# Run all baseline experiments
for script in script/rerank/baseline/*.sh; do
  bash "$script"
done

Results will be generated in the res/ directory with detailed evaluation metrics.

License

This project is provided for research purposes. Please refer to individual dataset licenses for usage terms.