Code to replicate CLIRudit: Cross-Lingual Information Retrieval of Scientific Documents (Valentini et al., MRL 2025).

Cite as:

@inproceedings{valentini2025clirudit,
    title = "{CLIR}udit: Cross-Lingual Information Retrieval of Scientific Documents",
    author = "Valentini, Francisco  and
      Kozlowski, Diego  and
      Lariviere, Vincent",
    editor = "Adelani, David Ifeoluwa  and
      Arnett, Catherine  and
      Ataman, Duygu  and
      Chang, Tyler A.  and
      Gonen, Hila  and
      Raja, Rahul  and
      Schmidt, Fabian  and
      Stap, David  and
      Wang, Jiayi",
    booktitle = "Proceedings of the 5th Workshop on Multilingual Representation Learning (MRL 2025)",
    month = nov,
    year = "2025",
    address = "Suzhuo, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.mrl-main.16/",
    doi = "10.18653/v1/2025.mrl-main.16",
    pages = "226--242",
    ISBN = "979-8-89176-345-6"
}

Setup

Cloning the repo with submodules: git clone --recurse-submodules https://github.com/ftvalentini/academic-clir.git.

It is recommended to use an environment to run the scripts and install the dependencies. To install conda, see below.

conda create -n academic-clir-mt python=3.11 -y &&
conda activate academic-clir-mt &&
conda install pytorch pytorch-cuda=12.4 -c pytorch -c nvidia -y &&
pip install -r requirements.txt
# for pytorch, choose cuda version according to nvidia-smi & https://pytorch.org/get-started/locally/
# conda install xformers -c xformers --solver=classic # for gte-multilingual-base NOTE not used
python -m spacy download en_core_web_sm
pip install git+https://github.com/EleanorJiang/BlonDe

For pyserini (BM25) we will unfortunately use a separate conda environment (best way it worked for us):

conda create -n pyserini python=3.10 -y
conda activate pyserini &&
conda install -c conda-forge openjdk=21 maven -y &&
conda install -c conda-forge lightgbm nmslib -y &&
conda install -c pytorch faiss-cpu pytorch -y &&
pip install pyserini
pip install fire

The same for SPLADE, PLAID-X, PSQ:

cd splade && conda env create -f conda_splade_env.yml
pip install --upgrade hydra-core
# # Add submodule if needed:
# git submodule add https://github.com/ftvalentini/splade.git splade

# # Add submodule if needed:
# git submodule add https://github.com/ftvalentini/ColBERT-X.git ColBERT-X
conda create -n plaidx python=3.10 -y &&
conda activate plaidx &&
cd ColBERT-X && pip install -e .[gpu] && cd .. &&
conda install --force-reinstall -c nvidia cuda-toolkit=12.6
# NOTE you need nvcc with same version as torch.version.cuda (python -c "import torch; print(torch.version.cuda)" -- '12.6')

# pip install faiss-gpu
conda create -n psq python=3.10 -y &&
conda activate psq &&
pip install fast_psq

And yet another env for COMET (MT evaluation):

git submodule add https://github.com/amazon-science/doc-mt-metrics.git comet
cd comet/COMET
conda create -n comet python=3.9 && conda activate comet
pip install -r requirements.txt && pip install -e .

Set API keys as environment variables:

echo "export OPENAI_ACADEMIC_CLIR_API_KEY='...'" >> ~/.bashrc
source ~/.bashrc

Others:

mkdir -p data
mkdir -p logs
chmod +x scripts/*.sh

Data

Downloading

We downloaded all erudit folders using globus.

Then:

zip -r raw_erudit.zip erudit_corpus # in server
scp -r beluga:/my/dir/raw_erudit.zip data/ # in local
unzip data/raw_erudit.zip -d data/ # in local

Counting in globus, there are 326 files in the whole erudit corpus (325 dirs and one file) i.e. there are 325 journals.

Making the dataset

1. Run to extract the raw corpus from the XML files:

python scripts/extract_erudit_corpus.py

See details of the documents' structure below.

2. To scrape the disciplines of the journals from the Erudit website:

python scripts/scrape_erudit_disciplines.py --out_file "runs/erudit_corpus/journals_disciplines.pkl"

3. Convert the raw corpus into an IR dataset:

date=$(date '+%d-%m-%Y_%H-%M-%S') && \
nohup nice -n 19 scripts/wrapper_make_ir_datasets.sh \
    > logs/wrapper_make_ir_datasets_$date.log 2>&1 &
    # NOTE the generated qrels file includes some cols which are not usually included
    # but are useful for exlporatory analysis

We uploaded the final dataset to HuggingFace with notebooks/upload_dataset_to_hf.ipynb.

Experiments

See run.sh.

Then make plots and results with:

• notebooks/eval_table.ipynb
• notebooks/eval_mt.ipynb
• notebooks/eval_conf_intervals.ipynb
• notebooks/error_analysis.ipynb
• notebooks/eval_plots_disciplines.ipynb
• notebooks/eval_plots.ipynb
• notebooks/explore_ir_dataset.ipynb

Useful resources

• https://datasets.docs.erudit.org/i18n/en/datasets/erudit.html
• https://docs.globus.org/api/transfer/overview/
• https://github.com/castorini/pyserini/blob/master/docs/installation.md#linux
• https://github.com/naver/splade

Installing Conda

mkdir -p ~/miniconda3 &&
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh &&
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3 &&
rm -rf ~/miniconda3/miniconda.sh &
~/miniconda3/bin/conda init bash &&
conda list

Structure of the documents in Erudit

Dir structure to parse:

erudit
├── journal directory (journal_id)
│   ├── LOGO
│   ├── PUBLICATIONS.xml
│   ├── THEMES.xml
│   └── issue directory (issue_id)
|       ├── COVERPAGE_HD *
|       ├── COVERPAGE *
|       ├── SUMMARY.xml
|       └── article directory (article_id)
|           ├── ERUDITXSD300.xml
|           ├── PDF.pdf *
|           ├── INFOIMG.xml *
|           └── assets directory *
|               └── asset files *

• From dir structure, should get journal_id (325 total).

• From PUBLICATIONS.xml, should get:

<!-- issue_id, issue_year, issue_period, issue_number, issue_volume -->
<numero annee="2011" autrenumero="" nonumero="51" periode="Automne" pid="erudit:erudit.crs123.crs0516" revAbr="crs" type="reg" url="crs/2011/v/n51" volume=""></numero>

• From SUMMARY.xml, should get:

<!-- journal_title, journal_id, journal_lang -->
<revue id="crs123" lang="fr">
    <titrerev>Cahiers de recherche sociologique</titrerev>
    <titrerevabr>...</titrerevabr>
    ...
</revue>

<!-- issue_id, issue_period, issue_year, issue_number, issue_theme -->
  <numero id="crs0516">
    <nonumero>51</nonumero>
    <pub>
      <periode>Automne</periode>
      <annee>2011</annee>
    </pub>
    <pubnum>
      <date typedate="publication">2013-03-26</date>
    </pubnum>
    <grtheme id="th1">
          <theme>Théâtralité et société : positions de la sociologie</theme>
    </grtheme>
    ...
  </numero>

<!-- issue_editor -->
<editeur>
  <nomorg>Athéna éditions</nomorg>
  <key>editeur</key>
</editeur>

<!-- article_id, article_lang, article_type, article_number, article_title (each lang), article_abstract (each lang), article_keywords (each lang) -->
<article doi="10.7202/1014995ar" idproprio="1014995ar" idref="th1"
    lang="fr" ordseq="2" qualtraitement="minimal" typeart="article">
    ...
    <grtitre>
        <titre>Le mouvement, matière du drame et de l’épopée</titre>
        <titreparal lang="en">Movement: Substance of Drama and Epic</titreparal>
        <titreparal lang="es">El movimiento, materia del drama y de la epopeya</titreparal>
    </grtitre>
    ...
    <resume typeresume="resume" lang="fr">
        <alinea>...</alinea>
    </resume>
      <resume typeresume="resume" lang="en">
        <alinea>...</alinea>
    </resume>
  ...
    <grmotcle lang="fr">
      <motcle>...</motcle>
      <motcle>...</motcle>
      ...
    </grmotcle>
    <grmotcle lang="en">
      <motcle>...</motcle>
      <motcle>...</motcle>
      ...
    </grmotcle>