X-RAG: Explainable Retrieval-Augmented Generation via Argumentation Frameworks and Dempster-Shafer Theory

X-RAG is a retrieval-augmented reasoning framework that keeps the LLM only as a relation extractor and delegates all conflict resolution to two mathematically grounded, fully auditable components: Dempster-Shafer Theory (DST) for uncertainty-aware evidence fusion and a Quantitative Bipolar Argumentation Framework (QBAF) for structured conflict resolution.

The motivation is a failure mode we call majority-vote collapse: when a standard RAG system aggregates conflicting retrieved documents by majority vote, adding more documents makes it worse, and on Adversarial NLI it collapses to predicting neutral for everything (34% accuracy, 0% attack precision at K of 3 and 5). X-RAG holds 57 to 69% under the same conditions and produces a per-edge, fully inspectable reasoning trace.

A step-by-step explanation (in Vietnamese) is available in docs/X-RAG_explained_vi.md.

---

Pipeline overview

The framework runs in three phases, mirrored by the source layout:

---

Repository structure

X-RAG/
├── src/xrag/
│   ├── phase1_relation_extraction/
│   │   ├── retriever.py            # ChromaRetriever (vector search over the KB)
│   │   └── relation_extractor.py   # RelationExtractor (LLM judge, constrained JSON)
│   ├── phase2_dst_fusion/
│   │   └── dst_engine.py           # DSTEngine (mass functions, Dempster's rule, Belief)
│   ├── phase3_qbaf/
│   │   └── arg_engine.py           # QBAFSolver (Quadratic Energy gradual semantics)
│   └── pipeline/
│       ├── reasoning_pipeline.py   # DST + QBAF (deterministic core, NumPy only)
│       └── xrag_system.py          # full end-to-end pipeline
├── experiments/                    # evaluation + ablation scripts
│   └── results/                    # metrics & per-instance predictions (JSON) from the paper
├── figures/                        # figure-generation scripts and rendered figures
├── data/                           # dataset instructions (ANLI is downloaded, not stored)
├── docs/                           # Vietnamese explainer
├── pyproject.toml
├── requirements.txt
└── .env.example

---

Installation

git clone <your-public-repo-url> X-RAG
cd X-RAG
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
pip install -e .          # makes `import xrag...` work everywhere

For remote models (GPT-4o-mini via OpenRouter), copy .env.example to .env and set your key:

cp .env.example .env      # then edit OPENROUTER_API_KEY

Local open models are served with Ollama (e.g. ollama pull gemma2, nomic-embed-text).

---

Dataset

X-RAG is evaluated on ANLI (Adversarial NLI), a public 3-class benchmark.

• HuggingFace: https://huggingface.co/datasets/facebook/anli
• Source (Nie et al., ACL 2020): https://github.com/facebookresearch/anli

Loading (used by the experiment scripts):

from datasets import load_dataset
ds = load_dataset("anli", split="dev_r1")   # rounds: dev_r1 / dev_r2 / dev_r3

We sample 100 instances per round (R1, R2, R3) at the 33/33/34 class balance, with the mapping entailment → support, neutral → neutral, contradiction → attack. See data/README.md.

---

Quickstart

The deterministic core (DST + QBAF) needs only NumPy and no LLM:

from xrag.pipeline.reasoning_pipeline import XRAGReasoningPipeline

pipeline = XRAGReasoningPipeline()
docs = [
    {"id": "Doc1", "similarity": 0.90, "label": "S"},   # supports the claim
    {"id": "Doc2", "similarity": 0.85, "label": "C"},   # contradicts (attacks) the claim
]
edges = [
    {"from": "Doc1", "to": "Claim", "weight": 0.8, "type": "support"},
    {"from": "Doc2", "to": "Claim", "weight": 0.8, "type": "attack"},
]
strengths = pipeline.run_pipeline("Claim", docs, edges)
print(strengths["Claim"])   # converged claim strength σ(C) ∈ [0,1]

Full end-to-end system (requires Ollama + ChromaDB):

from xrag.pipeline.xrag_system import XRAGSystem
sys = XRAGSystem()
sys.ingest([{"id": "d1", "text": "..."}, {"id": "d2", "text": "..."}])
answer, score = sys.query("Your hypothesis here", top_k=3)

Note: in the code the DST frame uses labels S (Support) and C (Contradict). C corresponds to the Attack class used in the paper.

---

Reproducing the experiments

# Relation-extractor / baseline NLI accuracy per model and round
python experiments/eval_relation_extractor.py
python experiments/eval_openrouter.py --round r1 --samples 100     # GPT-4o-mini via OpenRouter

# End-to-end ablation (Standard RAG / QBAF-only / DST-only / X-RAG) across K
python experiments/eval_xrag_e2e.py

# Synthetic conflict-resolution study
python experiments/eval_synthetic_graphs.py

# Significance tests and result compilation
python experiments/compute_pvalues.py
python experiments/compile_results.py

Pre-computed metrics and per-instance predictions from the paper are in experiments/results/*.json.

---

Key results

On ANLI (100 instances per split):

• DST is the primary noise-suppressor: +23% over Standard RAG at K=3, with attack precision 1.00 (statistically significant, McNemar p < 0.001).
• QBAF is a precise, non-destructive corrector: it changes few verdicts, but every change moves toward the gold label (e.g. 2 of 100 flips on GPT-4o-mini, all wrong→correct).
• On controlled synthetic conflicts, a no-cross-edge baseline scores 0% while X-RAG scores 100% on balanced and minority-support cases.

Figures are in figures/ (regenerate with the figures/gen_*.py scripts).

---

Limitations

• Evaluated on ANLI only (100 per split); this is an NLI-based proxy for multi-document RAG.
• The DST similarity tiers (0.5 / 0.8), decision band (0.4 / 0.6), and confidence clip are fixed defaults; a sensitivity sweep is left for future work.
• Cross-document extraction costs O(K²) LLM calls, reducible with a top-m approximation for large K.

---

Citation

Citation details will be added once the paper has been presented and published.

License

Released under the MIT License (see LICENSE).

Contact

Quang-Dung Dang — [email protected] — Data Governance Laboratory, Posts and Telecommunications Institute of Technology (PTIT), Hanoi, Vietnam.