STaRK

Website | Github | Paper

STaRK is a large-scale semi-structure retrieval benchmark on Textual and Relational Knowledge Bases

Downstream Task

Retrieval systems driven by LLMs are tasked with extracting relevant answers from a knowledge base in response to user queries. Each knowledge base is semi-structured, featuring large-scale relational data among entities and comprehensive textual information for each entity. We have constructed three knowledge bases: Amazon SKB (ecommence knowledge base), MAG SKB (academic database), and Prime SKB (biomedicine knowledge base).

Benchmarking

To evaluate model performance on SKB retrieval tasks, STaRK includes:

1. Synthesized queries that simulate real-world user requests,
2. Human-generated queries for authentic benchmarks and evaluation,
3. Precisely verified ground truth answers/nodes through automatic and manual filtering

The STaRK benchmark features three novel retrieval-based question-answering datasets, each containing synthesized train/val/test sets with 9k to 14k queries and a high-quality, human-generated query set. These queries integrate relational and textual knowledge, closely resembling real-world queries with their natural-sounding language and flexible formats.

Citation Information

@article{wu24stark,
    title        = {STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases},
    author       = {
        Shirley Wu and Shiyu Zhao and 
        Michihiro Yasunaga and Kexin Huang and 
        Kaidi Cao and Qian Huang and 
        Vassilis N. Ioannidis and Karthik Subbian and 
        James Zou and Jure Leskovec
    },
    eprinttype   = {arXiv},
    eprint       = {2404.13207},
  year           = {2024}
}