More details on the dataset can be found in our paper: https://arxiv.org/abs/2412.17031.
DRUID contains real-world (query, context) pairs to facilitate studies of context usage and failures in real-world RAG scenarios. The dataset is based on the prototypical task of automated claim verification, for which automated retrieval of real-world evidence is crucial. Therefore, we sometimes also refer to 'query' as 'claim' and 'context' as 'evidence'.
We release two versions of the dataset: DRUID and DRUID+. DRUID is a high-quality (corresponding to top Cohere reranker scores) subset of DRUID+ manually annotated for evidence relevance and stance. The dataset contains the following columns:
Samples based on claims from 'borderlines' follow a sligthly different structure, as the "claims" in this case are based on the borderlines dataset. Therefore, they have no corresponding claimant, claim source or fact-check verdict, etc.
We sample claims verified by fact-checkers using Google's Factcheck API. We only sample claims in English. The claims are collected from 7 diverse fact-checking sources, representing science, politics, Northern Ireland, Sri Lanka, the US, India, France, etc. All claims have been assessed by human fact-checkers. We also collect claims based on the borderlines dataset, for which we can expect there to be more frequent conflicts between contexts.
For each claim in DRUID DRUID+, we retrieve up to 5 and 40 snippets of evidence, respectively. First, a gold-standard evidence document is retrieved from the original fact-checking site, which is the 'summary' of the fact-checking article written by the author of the article. For the remaining snippets of evidence, we use an automated retrieval method. We collect the top 20 search results for each of the Google and Bing search engines. The found webpages are then chunked into paragraphs and reranked by the Cohere rerank model (`rerank-english-v3.0' from here). Evidence corresponding to the top-ranked chunks is included in DRUID.
Since the evidence is collected using automated retrieval, as opposed to controlled synthesis, we need to assess the relevance of the retrieved information to the claim, and, if it is relevant, what stance it represents. For this, we crowd-source evidence-level annotations using Prolific and Potato. Each evidence piece in DRUID is double annotated for relevance (relevant or not relevant) and stance to the claim (supports, insufficient-supports, insufficient-neutral, insufficient-contradictory, insufficient-refutes or refutes).
The annotator compensation was approximately 9 GBP/hour (the compensation was fixed for each task while the annotator completion time varied).
@misc{druid,
title={A Reality Check on Context Utilisation for Retrieval-Augmented Generation},
author={Lovisa Hagström and Sara Vera Marjanović and Haeun Yu and Arnav Arora and Christina Lioma and Maria Maistro and Pepa Atanasova and Isabelle Augenstein},
year={2024},
eprint={2412.17031},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2412.17031},
}
7 commits
More details on the dataset can be found in our paper: https://arxiv.org/abs/2412.17031.
DRUID contains real-world (query, context) pairs to facilitate studies of context usage and failures in real-world RAG scenarios. The dataset is based on the prototypical task of automated claim verification, for which automated retrieval of real-world evidence is crucial. Therefore, we sometimes also refer to 'query' as 'claim' and 'context' as 'evidence'.
We release two versions of the dataset: DRUID and DRUID+. DRUID is a high-quality (corresponding to top Cohere reranker scores) subset of DRUID+ manually annotated for evidence relevance and stance. The dataset contains the following columns:
Samples based on claims from 'borderlines' follow a sligthly different structure, as the "claims" in this case are based on the borderlines dataset. Therefore, they have no corresponding claimant, claim source or fact-check verdict, etc.
We sample claims verified by fact-checkers using Google's Factcheck API. We only sample claims in English. The claims are collected from 7 diverse fact-checking sources, representing science, politics, Northern Ireland, Sri Lanka, the US, India, France, etc. All claims have been assessed by human fact-checkers. We also collect claims based on the borderlines dataset, for which we can expect there to be more frequent conflicts between contexts.
For each claim in DRUID DRUID+, we retrieve up to 5 and 40 snippets of evidence, respectively. First, a gold-standard evidence document is retrieved from the original fact-checking site, which is the 'summary' of the fact-checking article written by the author of the article. For the remaining snippets of evidence, we use an automated retrieval method. We collect the top 20 search results for each of the Google and Bing search engines. The found webpages are then chunked into paragraphs and reranked by the Cohere rerank model (`rerank-english-v3.0' from here). Evidence corresponding to the top-ranked chunks is included in DRUID.
Since the evidence is collected using automated retrieval, as opposed to controlled synthesis, we need to assess the relevance of the retrieved information to the claim, and, if it is relevant, what stance it represents. For this, we crowd-source evidence-level annotations using Prolific and Potato. Each evidence piece in DRUID is double annotated for relevance (relevant or not relevant) and stance to the claim (supports, insufficient-supports, insufficient-neutral, insufficient-contradictory, insufficient-refutes or refutes).
The annotator compensation was approximately 9 GBP/hour (the compensation was fixed for each task while the annotator completion time varied).
@misc{druid,
title={A Reality Check on Context Utilisation for Retrieval-Augmented Generation},
author={Lovisa Hagström and Sara Vera Marjanović and Haeun Yu and Arnav Arora and Christina Lioma and Maria Maistro and Pepa Atanasova and Isabelle Augenstein},
year={2024},
eprint={2412.17031},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2412.17031},
}
7 commits