README
ParaConflict – Parametric-Knowledge Conflict Evaluation Set
(dataset used in “Taming Knowledge Conflicts in Language Models”)
What’s included
| Field | Type | Description |
|---|---|---|
Subject | str | Entity whose factual attribute is tested — e.g. "France" |
Answer | List[str] | Gold fact(s) from a curated KB — e.g. ["Paris"] |
Distractor_Token | str | Token injected to create a conflict |
Clean_Prompt | str | Canonical question with no conflict |
Substitution_Conflict | str | Same prompt but with the gold answer replaced by the distractor |
Coherent_Conflict | str | Longer fluent passage embedding the same false claim |
Intended use
- Provide a clean, controlled benchmark for assessing LLM robustness when contextual input contradicts their parametric knowledge under different degrees.
- Contexts are treated as misleading information in this dataset.
Code
Github
Quick start (🤗 Datasets)
from datasets import load_dataset
ds = load_dataset("gaotang/ParaConflict", split="test")
print(ds[0])
Similar repos
README
ParaConflict – Parametric-Knowledge Conflict Evaluation Set
(dataset used in “Taming Knowledge Conflicts in Language Models”)
What’s included
| Field | Type | Description |
|---|---|---|
Subject | str | Entity whose factual attribute is tested — e.g. "France" |
Answer | List[str] | Gold fact(s) from a curated KB — e.g. ["Paris"] |
Distractor_Token | str | Token injected to create a conflict |
Clean_Prompt | str | Canonical question with no conflict |
Substitution_Conflict | str | Same prompt but with the gold answer replaced by the distractor |
Coherent_Conflict | str | Longer fluent passage embedding the same false claim |
Intended use
- Provide a clean, controlled benchmark for assessing LLM robustness when contextual input contradicts their parametric knowledge under different degrees.
- Contexts are treated as misleading information in this dataset.
Code
Github
Quick start (🤗 Datasets)
from datasets import load_dataset
ds = load_dataset("gaotang/ParaConflict", split="test")
print(ds[0])