INTER-PASSAGE-LSR

This guide provides a step-by-step walkthrough to reproduce the results of our project from scratch. It covers data setup, environment configuration, model training, evaluation, and result analysis. The following images present an overview of our novel Dependent Score Max and Passage Block methods.

---

🧱 Project Structure

.
├── figures/                # Figures used in the paper
├── data/                   # Raw and processed datasets
├── src/                    # Source code (models, training, evaluation)
├── slurm_script/           # SLURM scripts for distributed computing
├── requirements.txt        # Python dependencies
├── README.md               # Project documentation

---

⚙️ Environment Setup

To set up the environment, run the following SLURM job:

slurm_script/cpu/install_environment.job

Alternatively, you may manually execute the commands contained in the job script.

---

📂 Download & Prepare Datasets

Follow these steps to prepare the datasets:

1. Download Datasets Place all datasets in the data/ directory. Use the relevant SLURM job based on the task:

   slurm_script/cpu/preprocess/download_dataset/{task}.job
   

   Supported tasks:

   • passage
   • passage_with_title (requires passage dataset)
   • multi_psgs (based on lsr-long; filters out conflicting positives/negatives)

2. Split Documents for Segmentation Use the following jobs based on the model and token window size:

   slurm_script/cpu/preprocess/split_msmarco_doc/
   

3. Generate GB Candidate Lists Candidate generation requires pre-downloaded datasets from Step 1:

   slurm_script/cpu/preprocess/gb_candidate_list/{task}.job
   

---

⚙️ Configuration

Configuration files are located in:

lsr/config/

Each experiment has a structured configuration path:

lsr/config/experiment/{operation}/{task}/{method}/[dataset_if_evaluation]

Where:

• operation: training or evaluation
• task: multi_psgs, psgs, psg_with_title
• method: Specific approach or model variant
• dataset: Optional, used for evaluation-specific configs

---

🚀 Training

To launch a training job on H100 GPUs:

slurm_script/gpu_h100/train/{task}/{method}.job

Note: If using different hardware, adjust the job script and configurations accordingly.

---

📈 Evaluation

After training, follow these steps for evaluation:

1. Update the experiment config under:

   lsr/config/experiment/evaluation/{task}/{method}/{dataset}
   

   Ensure it points to the correct checkpoint.

2. Run the evaluation job on A100 GPUs:

   slurm_script/gpu_a100/eval/{task}/{method}/{dataset}.job
   

Note: Modify configurations if using hardware other than A100.

---

📊 Analysis

There are three types of analysis:

1. Analysis Using CSV Files

• Download the relevant CSVs from Weights & Biases (wandb) or use provided ones.

• Run the analysis job:

  slurm_script/cpu/analysis/{task}.job
  

2. Analysis Independent of Run

• Simply run the job script:

  slurm_script/cpu/analysis/{task}.job
  

3. Inference-Based Analysis

• Set the appropriate configuration.

• Execute the inference analysis:

  slurm_script/gpu_a100/analysis/{task}.job
  

---

This repository was based on the following repository:

• https://github.com/thongnt99/learned-sparse-retrieval (The format)
• https://github.com/thongnt99/lsr-long (How to handle long documents)
• https://github.com/webis-de/set-encoder (How to implement the INT token)