---

Overview of CausalAbstain for trustworthy abstention using native-language and multilingual feedback.

🔍 Overview

Large Language Models (LLMs) exhibit substantial knowledge disparities across languages. A model may confidently answer a question in one language while lacking sufficient knowledge to answer the same question reliably in another.

A natural way to reduce hallucinations is to let the model abstain when it is uncertain. Existing multilingual abstention methods often rely on self-reflection or feedback generated in different languages. However, generated feedback itself can be noisy, biased, or incorrect.

CausalAbstain addresses this problem from a causal perspective:

Instead of trusting generated feedback by default, CausalAbstain estimates whether the feedback actually contributes useful information to the final abstention decision.

The framework measures the effect of answering without feedback and compares it against the effect mediated through feedback. This allows the model to decide both:

• whether feedback should be used, and
• which feedback is sufficiently informative to influence the abstention decision.

CausalAbstain supports two settings:

• CAUSAL-NATIVE — uses feedback generated in the question's native language.
• CAUSAL-MULTI — considers feedback from multiple related languages and aggregates useful multilingual signals.

Experiments on MMLU and HellaSwag show that CausalAbstain improves trustworthy abstention over strong baselines across multilingual knowledge and commonsense reasoning settings.

---

🚀 Quick Start

1. Installation

Clone the repository:

git clone https://github.com/peachch/CausalAbstain.git
cd CausalAbstain

Create the provided Conda environment:

conda env create -f causalabstain.yaml
conda activate causal

---

2. Running Local Models with Ollama

The released implementation supports local Llama and Phi inference through Ollama.

Install Ollama:

curl -fsSL https://ollama.com/install.sh | sh

Start the Ollama server:

export OLLAMA_HOST="127.0.0.1:8000"
ollama serve

Pull or run a model, for example:

ollama run llama3.2

For Phi:

ollama run phi4

---

3. API-Based Models

Model-specific inference logic is implemented in:

lm_utils.py

For API-based models, configure the corresponding endpoint and credentials in the model backend before running the experiment.

If you add another LLM provider, the main interface to extend is:

llm_response(...)

in lm_utils.py.

---

🧪 Running CausalAbstain

A typical multilingual run using an API-based model:

python causalabstain.py \
    -m gpt3.5 \
    -d mmlu \
    -s zh \
    -l true \
    -r three \
    -n 3 \
    -f True \
    -t test \
    -o 0.5

Using a local Llama model:

python causalabstain.py \
    -m llama \
    -d mmlu \
    -s it \
    -l true \
    -r three \
    -n 3 \
    -f True \
    -t test \
    -o 0.5

Main Arguments

For example, to evaluate half of the Italian MMLU data with three related-language feedback sources:

python causalabstain.py \
    --model llama \
    --dataset mmlu \
    --speak it \
    --related three \
    --iter_number 3 \
    --portion 0.5

---

📚 Supported Datasets

The released repository contains multilingual versions of two benchmarks:

Dataset files follow the structure:

data/
├── mmlu/
│   ├── mmlu_<language>.json
│   └── ...
│
└── hellaswag/
    ├── hellaswag_<language>.json
    └── ...

The --speak argument selects the corresponding language file.

---

📊 Baselines

Baseline implementations are provided under:

baselines/

The released repository includes:

baselines/
├── approach-askcalibrate.py
├── approach-conflict.py
├── approach-moreinfo.py
├── approach-multirelated.py
├── approach-reflect.py
└── baselines.md

These implementations follow the multilingual abstention comparison setup used in the paper.

A baseline can be run, for example, with:

cd baselines

python approach-conflict.py \
    -m llama \
    -d mmlu \
    -s kn \
    -l True \
    -o 0.5

See baselines/baselines.md for additional details.

---

📂 Repository Structure

CausalAbstain/
├── README.md
├── causalabstain.py              # Main CausalAbstain pipeline
├── causalabstain.yaml            # Conda environment
├── lm_utils.py                   # LLM inference, distributions, and JSD
├── metrics.py                    # Abstention evaluation metrics
│
├── baselines/                    # Baseline methods
│   ├── approach-askcalibrate.py
│   ├── approach-conflict.py
│   ├── approach-moreinfo.py
│   ├── approach-multirelated.py
│   ├── approach-reflect.py
│   └── baselines.md
│
├── data/                         # Multilingual evaluation datasets
│   ├── mmlu/
│   └── hellaswag/
│
└── imgs/
    └── multilingual_combine (1).png

Core Components

causalabstain.py

Implements the main CausalAbstain workflow:

• obtains the model's proposed answer;
• estimates abstention behavior without feedback;
• generates feedback in related languages;
• estimates feedback-mediated effects;
• compares NDE and TIE signals;
• aggregates multilingual feedback;
• produces the final abstain / not-abstain decision.

lm_utils.py

Provides:

• LLM inference;
• local Ollama integration;
• response parsing;
• abstention-distribution construction;
• Jensen–Shannon Divergence computation.

metrics.py

Implements the evaluation metrics for trustworthy abstention.

---

📜 Citation

CausalAbstain was published in Findings of ACL 2025.

If you find this work useful, please cite:

@inproceedings{sun-etal-2025-causalabstain,
    title = "{C}ausal{A}bstain: Enhancing Multilingual {LLM}s with Causal Reasoning for Trustworthy Abstention",
    author = "Sun, Yuxi and Zuo, Aoqi and Gao, Wei and Ma, Jing",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.findings-acl.723/",
    doi = "10.18653/v1/2025.findings-acl.723",
    pages = "14060--14076"
}