HaluBench is a hallucination evaluation benchmark of 15k samples that consists of Context-Question-Answer triplets annotated for whether the examples contain hallucinations. Compared to prior datasets, HaluBench is the first open-source benchmark containing hallucination tasks sourced from real-world domains that include finance and medicine.
We sourced examples from several existing QA datasets to build the hallucination evaluation benchmark. We constructed tuples of (question, context, answer, label), where label is a binary score that denotes whether the answer contains a hallucination.
The examples are sourced from and constructed using existing datasets such as FinanceBench, PubmedQA, CovidQA, HaluEval, DROP and RAGTruth.
HaluBench can be used to evaluate hallucination detection models. The PatronusAI/Llama-3-Patronus-Lynx-70B-Instruct outperforms GPT-4o, Claude-Sonnet and other open source models on HaluBench. PatronusAI/Llama-3-Patronus-Lynx-8B-Instruct is a 8B variant that has only a ~3% gap compared to GPT-4o.
HaluBench is a hallucination evaluation benchmark of 15k samples that consists of Context-Question-Answer triplets annotated for whether the examples contain hallucinations. Compared to prior datasets, HaluBench is the first open-source benchmark containing hallucination tasks sourced from real-world domains that include finance and medicine.
We sourced examples from several existing QA datasets to build the hallucination evaluation benchmark. We constructed tuples of (question, context, answer, label), where label is a binary score that denotes whether the answer contains a hallucination.
The examples are sourced from and constructed using existing datasets such as FinanceBench, PubmedQA, CovidQA, HaluEval, DROP and RAGTruth.
HaluBench can be used to evaluate hallucination detection models. The PatronusAI/Llama-3-Patronus-Lynx-70B-Instruct outperforms GPT-4o, Claude-Sonnet and other open source models on HaluBench. PatronusAI/Llama-3-Patronus-Lynx-8B-Instruct is a 8B variant that has only a ~3% gap compared to GPT-4o.
