RAREval-data-processing

📘 Overview

This repository provides the official code for the data preprocessing procedures introduced in our paper:
Do Reviews Matter for Recommendations in the Era of Large Language Models?

The figure above presents the RAREval framework, which is designed to evaluate review-aware recommender systems under five distinct experimental conditions:

• No-Review: Removes all textual reviews
• Reduction: Randomly removes a portion of review texts (0%, 25%, 50%, 75%, 100%)
• Distortion: Reassigns review texts among user-item pairs (0%, 25%, 50%, 75%, 100%)
• Data Sparsity: Applies k-core filtering and splits into train/valid/test
• Cold-Start: Evaluates performance across user groups with 1–10 historical interactions (no preprocessing required)

This repository provides a suite of scripts that generate modified datasets corresponding to the above five evaluation settings, enabling reproducible benchmarking and robustness analysis of large language model-based recommendation systems.

📦 Requirements

Tested on:

📚 Core Libraries

🤗 HuggingFace & LLM Support

🧠 Model Runtime Support

---

📂 Dataset

You need to prepare the following datasets before running RAREval.

🔗 Download from: https://cseweb.ucsd.edu/~jmcauley/datasets/amazon/links.html

Please download the 5-core version for the following 8 domains into the data/ directory, and likewise, the raw reviews or 0-core version for the same 8 domains into the data/raw_reviews directory:

---

🔬 Experiment Examples

To run each RAREval experiment, first execute train_valid_test_split.sh to split the data into training, validation and test set, as follows

./train_valid_test_split.sh

Then, execute the corresponding scripts:

📌 No-Review

python no_review_processing.py

📌 Reduction

python reduction_processing.py

📌 Distortion

python distortion_processing.py

📌 Data Sparsity

python data_sparsity_kcore.py

📌 Zero-shot and Few-shot

Example usage (change dataset_dir and llm as appropriate):

python zero_shot.py --dataset_dir data/true-data/reviews_Musical_Instruments --llm qwen2.5:0.5b
python few_shot.py --dataset_dir data/true-data/reviews_Musical_Instruments --llm qwen2.5:0.5b
python zero_shot_slow.py --dataset_dir data/true-data/reviews_Musical_Instruments --llm qwen2.5:0.5b
python few_shot_slow.py --dataset_dir data/true-data/reviews_Musical_Instruments --llm qwen2.5:0.5b

The slow version is only recommended for large language models that are not available in a single GGUF file.

For our experiments, we ran the slow version for large language models containing at least 6 billion parameters, and the fast version otherwise.

The output will be a .jsonl file on the main project directory showing the user training frequency, the item training frequency, predicted rating, and actual rating. This can be used for cold start analysis, and does not require cold_start_evaluation.py.

📌 LLM Summarization

Example usage (change dataset_dir as appropriate):

python summarize_short.py --dataset_dir data/true-data/reviews_Musical_Instruments

The file summarize_short.py uses Llama 3.2 (1B) to extract important data from review text, before summarizing the extracted data in one sentence for use for fine-tuning.

There is a file extract_only.py with similar usage that only does the extraction without summarizing the extracted data in one sentence.

📌 LLM LoRA finetuning

Example usage (change parameters as appropriate - loss functions supported are mse and mae):

python llm_lora.py --dataset_dir data/true-data/reviews_Musical_Instruments --llm qwen2.5:0.5b --instance_id testing --batch_size 16 --gradient_accumulation_steps 4 --loss_function mse

If you are running multiple instances of llm_lora.py, please make sure instance_id is different to avoid overwriting the same save file while training different models.

📌 Cold-Start

This setting requires no preprocessing script. Cold-start evaluation is performed directly during model evaluation, by grouping test users according to their training interaction count (e.g., 1–10).

We provide a standalone evaluation script to compute MSE, MAE, and sample count for each user group based on their frequency in the training set.

📁 Evaluation script: cold_start_evaluation.py

An example output for the Musical Instruments domain is shown below:

# Output Summary (Musical Instruments)
Training Frequency 0:  MSE = nan,     MAE = nan,     Count = 0
Training Frequency 1:  MSE = 0.5833,  MAE = 0.4167,  Count = 12
...
Training Frequency 10: MSE = 1.7273, MAE = 0.7273,Count = 22

📄 Full output available in examples/cold_start_example.txt

⚠️ Note: The evaluation script does not perform model inference. It assumes predictions are already generated by your model and saved in JSONL format.