live-rag-2025

How to

[!WARNING]
Before running the following commands, as they use parallel for the parallelization, please install apt-get install parallel

Query Rewrite

This stage rewrites the given query to reform the query without typo. To run this, navigate to /src/query_rewrite, and modify input path to the correct input query path (it is currently ./final_test.jsonl) Then, the results with corrected query will be saved at ./corrected_question.jsonl. (Can modify at the code)

python autocorrect.py

Routing decision to use spell-corrected or original queries, merging answer files

To decide which queries(original or corrected) to use, levenshtein edit distance is measured over pairs. If the pair distance is larger than threshold(e.g: 0.35), the corrected version substitutes the original one. To run this, original query and corrected query path and output path is required.

python -m src.edit_distance.main --input_path ${input_query} --rewritten_path ${corrected_query} \
    --output_path ${output}

[!IMPORTANT] This outputs are same query format so that can later used to run the retrieval. You should move the output file to correct path to run following steps.

First-Stage retrieval

This pipeline incorporates sparse (opensearch) and dense (pinecone) indices. As sparse and dense both have their own strengths, we choose hybrid strategy to sum the score of both indices.

[!IMPORTANT] The following command takes one directory to generate ranking file, ${input_directory}/1_merged_retrieval_evaluation/ordered_rankings.jsonl The directory MUST have ${input_directory}/0_init/query.jsonl to initiate the command

sh ./scripts/firststage_retrieval_parallel.sh ${input_directory}

Harmful and Non-english document filtering

We also excluded harmful documents from the retrieved results. As harmful documents should not be considered as context for RAG, we tagged all the documents with meta-llama/Llama-Guard-3-1B, and removed the documents with harmful codes. (For harmful codes, S6 and S8, we neglected them as they are "Specialized Advice" and "Intellectual Property" which are less likely to be treated harmful.) To tag the harmful documents,

sh ./src/harmful_filter/filter_harmful_texts_hf.sh \
    ${input_document_path_jsonl} \
    ${output_harmful_document_tags_jsonl}

In addition, we also eliminated non-English documents, as they are not the target for this competition. To do so, we tagged the document's language with fasttext. To tag the document's language,

sh ./src/language_filter/detect_language.sh \
    ${input_document_path_jsonl} \
    ${output_language_tagged_document_jsonl}

Hybrid Retrieval

We merge sparse (bm25) and dense (the NovaSearch/jasper_en_vision_language_v1 model) top-1000 retrieval results. The merged results include about 1300 - 2000 top-k documents for each query, depending on how much overlap exists on the top-1000 retrieved passages from bm25 and jasper. The merging script can be found at: src/utils/merge_bm25_and_jasper.py (You can modify the input and output path from the code)

Reranking with rankt5 and provence

• RankT5

We use a pointwise reranker to re-rank top-2000 (maximum) passages from previous stage. We use the huggingface-ported RankT5 variant, with huggingface identifier of Soyoung97/RankT5-base. Code directory: src/retrieval/run_rankt5.py Usage:

CUDA_VISIBLE_DEVICES=0 python run_rankt5.py --input ${merged_firststage_retrieval_file_from_previous_stage} --output_dir ${directory_to_save_output} --top_k ${topk_candidates_to_rerank} --start ${start_index} --end ${end_index}

The batch size is adjusted for A6000 48GB server (approx.120). Batch size can be adjusted by the --bsize argument. We run multiple process, varying the start and end indices for efficient processing with multiple GPUs.

• Provence

We use a pointwise reranker & pruning model to re-rank top-100 passages and prune the document text that removes noisy and irrelevant sentences. The model here is naver/provence-reranker-debertav3-v1 in huggingface. Code directory: scripts/components/rerank_queries_provence.sh Usage:

CUDA_VISIBLE_DEVICES=0 scripts/components/rerank_queries_provence.sh ${input_query_directory} ${merged_reranking_previous_stage} ${directory_to_save_output}

Default batch size is 32 and this process takes 4-5 seconds for top-100 passages per query in average. We run this step after running RankT5 step, so that we run for same chunked reranking result files.

• Merging splits

After running provence, we merge the output and save to the designated path using the merge_prov_out.py function at src/utils/ directory.

Answer Generation with efficient groundedness verifier

This pipeline takes the retrieval results to generate the final answer. The answers are generated with multiple conditions, such as,

1. sliding window applied on top-k context
2. multiple prompts are applied After generating the answers, to choose the final answer, we employed the groundedness verifier utilizing the embedding model to compare the number of sentences from the answer that are close to the sentences from the given retrieved document context.

Finally, the output files are formatted for the submission and validated.

[!IMPORTANT]
The following command takes one directory to generate the final answer at ${input_directory}/5_submission/answers.jsonl The directory MUST have ${input_directory}/0_init/query.jsonl and ${input_directory}/1_final_rankings_raw/rankings.jsonl, which is a ranking result taken from the previous stage.

CUDA_VISIBLE_DEVICES=0 sh ./scripts/generation_parallel.sh ${input_directory}