Dataset Card for BEIR Benchmark

fiqa is one of the datasets from the Question Answering task within BEIR, measuring financial article retrieval for a given financial query.

Dataset Description

• Homepage: https://beir.ai
• Repository: https://beir.ai
• Paper: https://openreview.net/forum?id=wCu6T5xFjeJ
• Leaderboard: https://docs.google.com/spreadsheets/d/1L8aACyPaXrL8iEelJLGqlMqXKPX2oSP_R10pZoy77Ns
• Point of Contact: nandan.thakur@uwaterloo.ca

Dataset Summary

BEIR is a heterogeneous benchmark built from 18 diverse datasets representing 9 information retrieval tasks.

• Fact-checking: FEVER, Climate-FEVER, SciFact
• Question-Answering: NQ, HotpotQA, FiQA-2018
• Bio-Medical IR: TREC-COVID, BioASQ, NFCorpus
• News Retrieval: TREC-NEWS, Robust04
• Argument Retrieval: Touche-2020, ArguAna
• Duplicate Question Retrieval: Quora, CqaDupstack
• Citation-Prediction: SCIDOCS
• Tweet Retrieval: Signal-1M
• Entity Retrieval: DBPedia

Languages

All tasks are in English (en).

Dataset Structure

This dataset uses the standard BEIR retrieval layout and includes:

• corpus: one row per document with _id, title, text
• queries: one row per query with _id, title, text

Data Fields

• _id (string): unique identifier
• title (string): title (empty string when unavailable)
• text (string): document/query text

Data Instances

A high level example of any BEIR dataset:

corpus = {
    "doc1" : {
        "title": "Albert Einstein",
        "text": "Albert Einstein was a German-born theoretical physicist. who developed the theory of relativity, \
                 one of the two pillars of modern physics (alongside quantum mechanics). His work is also known for \
                 its influence on the philosophy of science. He is best known to the general public for his mass–energy \
                 equivalence formula E = mc2, which has been dubbed 'the world's most famous equation'. He received the 1921 \
                 Nobel Prize in Physics 'for his services to theoretical physics, and especially for his discovery of the law \
                 of the photoelectric effect', a pivotal step in the development of quantum theory."
        },
    "doc2" : {
        "title": "", # Keep title an empty string if not present
        "text": "Wheat beer is a top-fermented beer which is brewed with a large proportion of wheat relative to the amount of \
                 malted barley. The two main varieties are German Weißbier and Belgian witbier; other types include Lambic (made\
                 with wild yeast), Berliner Weisse (a cloudy, sour beer), and Gose (a sour, salty beer)."
    },
}

queries = {
    "q1" : "Who developed the mass-energy equivalence formula?",
    "q2" : "Which beer is brewed with a large proportion of wheat?"
}

qrels = {
    "q1" : {"doc1": 1},
    "q2" : {"doc2": 1},
}

FiQA Data Splits

BEIR Direct Download

You can also download BEIR datasets directly (without loading through Hugging Face datasets) using the links below.

Citation Information

@inproceedings{
thakur2021beir,
title={{BEIR}: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models},
author={Nandan Thakur and Nils Reimers and Andreas R{\"u}ckl{\'e} and Abhishek Srivastava and Iryna Gurevych},
booktitle={Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)},
year={2021},
url={https://openreview.net/forum?id=wCu6T5xFjeJ}
}