Amazon-reviews for LLM Rerankers

Two small sequential recommendation datasets built from the Amazon Reviews 2023 corpus, published on Hugging Face and free to use with no token required:

Each is a set of per-user chronological interaction sequences where every interaction carries a fixed 10-item candidate set (1 ground truth + 9 in-domain negatives) for ranking evaluation, plus the item titles/descriptions a text-based ranker needs. Built small on purpose — on the order of 100 users each — so scoring every interaction with an LLM stays affordable.

Load Datasets

No need to clone this repository — just pip install datasets (or polars + huggingface_hub) and point at the repo id above.

Interaction splits, via the datasets library:

from datasets import load_dataset

ds = load_dataset("sungjin-code/amazon-reviews-for-llm")
# ds["train"], ds["validation"], ds["test"]

row = ds["test"][0]
row["candidates"]              # 10 item_idx values to rank
row["item_idx"], row["gt_pos"] # the ground truth, and its index inside candidates

Everything else — users.parquet, items.parquet, items_text.parquet (the titles/descriptions a text ranker actually reads), and items_meta.parquet (price, ratings, categories, images, and the rest of each item's metadata) aren't part of the datasets split config above, since they're not interaction rows. Read them directly instead:

import polars as pl

repo = "sungjin-code/amazon-reviews-for-llm"
base = f"https://huggingface.co/datasets/{repo}/resolve/main/data"

items = pl.read_parquet(f"{base}/items_text.parquet")   # item_idx -> title, description
users = pl.read_parquet(f"{base}/users.parquet")
inter = pl.read_parquet(f"{base}/interactions.parquet") # every split, undivided

Or download every file locally in one shot. This repo's example/load_dataset.py pulls the full published tree via huggingface_hub and flattens it into the plain, one-parquet-per-file layout the rest of this document assumes:

python example/load_dataset.py                    # -> out/cross_domain
python example/load_dataset.py --dataset books     # -> out/books

What's in it

Eight parquet files per dataset, plus a ninth for books:

interactions.parquet schema

Exact row/user/item counts and per-domain breakdowns are recorded in each dataset's own generated README.md on its HF page (cross-domain / books) — the source of truth for a specific build's statistics, since it's written straight from the data at build time rather than maintained by hand.

Candidate sets and evaluation protocol

Every interaction carries a fixed candidate set for ranking evaluation: the interaction's own item, plus 9 negatives sampled from the same domain.

• Negatives exclude every item the user interacts with anywhere in their sequence, so all 10 candidates are distinct and the ground truth is never mislabeled as a negative.
• The ground truth sits at a uniformly random position, recorded in gt_pos: candidates[gt_pos] == item_idx. A ranker cannot exploit slot position.
• Sampling is seeded at build time, so it's reproducible but otherwise arbitrary.

Evaluation is ranking within that 10-item set: give a reranker the user's history and the 10 candidate titles, ask for a full ordering, and score where the true item landed. With exactly one positive per list, HR@k / NDCG@k / MRR all collapse to a function of the target's rank.

Split protocol

Each user's interactions are sorted oldest to newest and cut by position, independently per user:

• train = positions [0, floor(0.8n))
• valid = positions [floor(0.8n), floor(0.9n))
• test = positions [floor(0.9n), n)

Every user appears in all three splits — a per-user ratio split, not a global time cut — so there's no temporal leakage within a user, and no split is ever empty.

Known limitations

Both datasets are small by design, so per-interaction LLM inference over the whole test split stays affordable. That has one real consequence: item overlap is very low — nearly every item appears exactly once, so most valid/test targets never occur in train. Text-based rankers are unaffected (they read items_text.parquet directly, no learned embedding needed), but id-embedding baselines like SASRec or GRU4Rec cannot be meaningfully evaluated on either build as published.

Items whose source metadata carried no title were dropped before user selection, so every candidate a ranker sees carries real text — there's no blank candidate to spot by process of elimination.

Run the zero-shot reranker baseline

This repo ships a working baseline against the published data, in example/. Requires cloning this repo (the dataset itself does not):

uv sync && source .venv/bin/activate
cp .env.example .env                       # sets MODEL=Qwen/Qwen3-4B-GGUF

python example/load_dataset.py             # download the dataset, see above
python utils/download_model.py             # pull the checkpoint named by MODEL
python example/qwen.py --check             # confirm the checkpoint loads, and whether GPU offload is available
python example/qwen.py --limit 20          # smoke run
python example/qwen.py                     # full test split

The reranker runs on llama.cpp, which has a genuine CPU fallback -- --check never hard-fails for lack of a GPU. GPU offload is attempted best-effort via --n-gpu-layers (default: all layers); --check reports whether it's actually available. --data-dir switches between the cross-domain and books builds; --model points at any local checkpoint. Three baselines run alongside the LLM (random, popularity, word-overlap-with-no-model) so its number can be read against something, and the run reports how often the model's reply had to be repaired into a valid ranking — read that before the metrics.

License and attribution

Both datasets are MIT-licensed and derived from the Amazon Reviews 2023 corpus released by the McAuley Lab at UC San Diego (project page · source corpus). See each HF repo's LICENSE and README.md (cross-domain / books) for the full notice, citation, and provenance details; they're generated per dataset and are the authoritative copy.

Reproducing / building from scratch

Want to build one of these yourself, or a different slice of the same pipeline (other domains, more users, a wider time window)? See data.md. CLAUDE.md covers the deeper architecture and the invariants a change to the build must preserve.