Two names, one project. BangHallu is the name used in the paper and in the demo. Bhibranti (ভ্রান্তি, Bengali for "delusion") is the repository's codename. They are the same thing.

In plain words

Language models — the programs behind chatbots — sometimes give an answer that sounds completely natural but is simply untrue. That is called a hallucination. It is dangerous because nothing in the wording warns you: a fluent wrong answer looks exactly like a fluent right one, and a student asking a question usually cannot check it, because they are asking precisely because they don't know.

This project asks one narrow question, in Bengali:

Given a question, a proposed answer, and sometimes a passage of supporting text — is the answer trustworthy, or made up?

Think of a teacher marking a student's answer. Sometimes the teacher has the textbook page open (a passage); sometimes they must mark from memory. We build software that plays that teacher, and — the part that matters — we measure how much of its skill is real understanding and how much is a cheap trick.

What we built

• A corpus of 4,480 Bengali question pairs (8,960 answers). Each question has exactly one correct answer and one made-up answer. Humans checked the labels; two annotators agree at Cohen's κ = 0.865.
• A ladder of 20 systems, from a two-line "is the answer's text in the passage?" rule, through word-counting and neural networks, up to a fine-tuned BanglaBERT — all scored by one shared piece of code so none can be judged on its own favourable terms.
• An interactive demo where you type a question and answer and watch all the systems vote.

What we found, in four lines

1. A trivial string-matching rule already scores 0.823 — with no learning at all. Any paper that reports only an average on this kind of data is mostly reporting that shortcut.
2. So we define a hard subset by measurement (the ~29% of pairs where the shortcut fails) and judge everything there. Best rule: 0.591. Best system without pretraining: 0.610.
3. Fine-tuned BanglaBERT reaches 0.825 on the hard subset — the only system that clearly beats the rules.
4. A Bengali-only pretrained model beat two multilingual ones, and the gap between "same architecture, pretrained" and "same architecture, trained from scratch" (0.825 vs 0.433) is the clearest measurement here of what pretraining is worth.

Label convention — the reverse of most papers. 1 = faithful / correct, 0 = hallucinated / made up. The positive class is the truthful one, everywhere in this repository.

Contents

1. Background for newcomers
2. How the project fits together
3. The dataset
4. Preparing Bengali text for models
5. The models — the ladder, rung by rung
6. How we score things (and why you can trust the numbers)
7. Results
8. The interactive demo
9. Reproduce it yourself
10. Repository guide
11. FAQ and troubleshooting
12. Limitations, ethics and future work
13. Citation and licence

1. Background for newcomers

Skip this section if you already work in NLP.

1.1 What is a hallucination?

When a language model doesn't know something, it doesn't say so — it produces the most plausible-sounding text. Ask it what year a mosque was founded and it will give you a year, fluently and confidently, whether or not it is correct. That confident falsehood is a hallucination.

1.2 What exactly does this project decide?

For every item — one question plus one proposed answer — the system outputs one of two labels:

Formally, with $q$ = question, $a$ = proposed answer, $c$ = supporting passage (which may be absent):

$$f_\theta(q,,a,,c);\longrightarrow;{0,,1}$$

Three real examples show the range of difficulty (English glosses in brackets):

1.3 Glossary

1.4 The ground rules

These constraints hold for the entire project and are what make the numbers meaningful:

2. How the project fits together

flowchart LR
    RAW["Raw sources<br/>62,084 items"] --> CLEAN["Clean and<br/>de-duplicate<br/>56,480"]
    CLEAN --> PAIRS["Build pairs<br/>4,480 pairs"]
    PAIRS --> LABEL["Human labelling<br/>kappa = 0.865"]
    LABEL --> SPLIT["Split by pair<br/>train / dev / test"]
    SPLIT --> GATE{"Shortcut audit<br/>probe below 0.60?"}
    GATE -->|"yes: 0.514"| LADDER["Train the ladder<br/>20 systems"]
    LADDER --> SCORE["One shared scorer<br/>evaluate.py"]
    SCORE --> DEMO["Demo and<br/>paper tables"]

    style GATE fill:#fff3cd,stroke:#b58900,color:#000
    style DEMO fill:#0ca30c,color:#fff,stroke:none

Every stage is a separate script you can run on its own (python src/<name>.py --help).

3. The dataset

3.1 Where the text comes from — and how the pairs were made

We want to be exact about this, because it affects how the corpus should be interpreted.

• Passages come from Bengali Wikipedia (CC BY-SA 4.0), collected into plain-text files. They are the grounding text for the has-context items.
• Questions and answer material come from publicly circulated Bangladesh school (SSC/HSC) and Civil Service (BCS) question banks, chiefly for the no-context items.
• The question–answer pairs themselves — both the correct and the made-up answer — were constructed with LLM assistance from those source texts, not written from scratch by hand. The facts originate in the source text; the LLM turned it into paired items. Every record carries a provenance field saying how its text was produced.
• Humans then verified the labels. Two annotators independently labelled the whole test split and typed each error; disagreements were adjudicated by hand.
• An LLM was used only as a second opinion for quality control (src/llm_annotate.py), never to write a label. Letting an LLM write the labels would be circular: a detector trained to reproduce an LLM's blind spots measures nothing. The script physically cannot overwrite a label — where it disagrees, the row is flagged for a human.

3.2 From raw pool to corpus

The 180 removed pairs are kept in the files, marked excluded = true; src/splits.py drops them automatically — which is why all data must be loaded through it (see §10.4).

3.3 One record, explained

{
  "id": "bnp_004674_1",          // unique item id (…_0 = correct, …_1 = wrong, for this pair)
  "pair_id": "bnp_004674",       // both answers to one question share this
  "question": "২০১০ সালে ইংল্যান্ডের বাংলাদেশ সফরে টেস্টে সাকিব কত উইকেট নিয়ে …?",
  "context": "২০১০-এর ফেব্রুয়ারি-মার্চে দুই টেস্ট ও তিন ওয়ানডে খেলতে …",   // "" when there is no passage
  "candidate_answer": "১০ উইকেট",  // the answer being judged
  "reference_answer": "৯ উইকেট",   // the known-correct answer for this question
  "label": 0,                     // 1 = faithful, 0 = hallucinated
  "condition": "has_context",     // or "no_context"
  "difficulty": "easy",           // measured: does string matching separate this pair?
  "hallucination_type": "numeric",// what kind of mistake (only for label 0)
  "subject": "history",
  "excluded": false               // true = removed after human review
}

3.4 Size and composition

The classes are exactly balanced by construction (every question contributes one of each), so plain accuracy would be a misleading score — see §6.1.

3.5 Subjects

13 subjects, drawn from school and civil-service exam material. Nothing was dropped for being hard — law, science and BCS questions that need outside knowledge stay in, and we report on them honestly.

Answers are short (median 16 characters — often one name, date or number); questions have a median of 55 characters; passages a median of 259, up to 3,132. That shape drives a design decision in §4.4: when input is too long, cut the passage, never the answer.

3.6 Kinds of mistakes

Every wrong answer is labelled with the kind of mistake, so we can ask not just how often a model is wrong but where:

_{*Of the 1,233 wrong answers in dev + test. Types are annotated for dev, test and a 20% spot-check of train; per-type results are therefore reported on dev/test only.}

3.7 "Hard" is measured, not opinion

The most important design decision in the project is how difficulty is defined. We never ask a human whether a question feels hard. We ask a mechanical question:

Does plain string matching separate the two answers of this pair?

If the correct answer appears in the passage and the wrong one does not, the pair is easy — a trivial rule solves it. If the shortcut can't tell them apart, the pair is hard. About 29% of the corpus is hard.

flowchart TD
    P["A pair: correct answer + wrong answer,<br/>with a passage"] --> Q{"Does string matching<br/>separate them?"}
    Q -->|"Yes: one appears in the passage, the other doesn't"| E["EASY<br/>a trivial rule solves it"]
    Q -->|"No: both appear, or neither does"| H["HARD<br/>the system must actually do something more"]
    style E fill:#dbeafe,stroke:#2a78d6,color:#000
    style H fill:#fee2e2,stroke:#e34948,color:#000

The same no-learning rule, measured on each subset — this table is the reason the hard subset exists:

A model that scores 0.83 overall has not shown anything — it barely beats a string search. That is why this project's target is the hard subset.

3.8 Human annotation and agreement

• Two annotators labelled independently against a written guideline (docs/ANNOTATION_GUIDELINES.md).
• Pilot: 100 blind items → κ = 0.717. Every disagreement was examined and five extra rules were written to cover the cases that caused them (a guideline problem, not an annotator problem).
• Full test split: all 1,356 items double-annotated (not a sample) → κ = 0.865, 93.2% raw agreement.
• Adjudication: all 253 remaining disagreements were decided one by one.
• Exclusions: where review concluded the stored label was wrong or the item was broken, the whole pair was removed from training and scoring — 62 train, 54 dev, 64 test pairs. Another 222 records where a human disagrees with the stored label are recorded but not applied (data/annotated/round1/label_disputes.csv) — a finding about the corpus, not something silently "fixed".

3.9 Safety checks on the data itself

Before any model was trained, we checked that the label cannot be guessed from the answer's surface properties — its length, punctuation, digit share, script mix, capitalisation.

This is a gate: had it failed, the corpus itself would be broken and every later number meaningless. It must pass before anything is trained (python src/audit.py --data data/splits --probe metadata; exits non-zero on failure).

3.10 Licensing of the data

The MIT licence in this repository covers the code only. The text under data/ is derived from Bengali Wikipedia (CC BY-SA 4.0 — attribution and share-alike apply) and from BCS/school question banks whose redistribution terms have not been confirmed. The full register, including the open questions, is in data/SOURCES.md. Please read it before reusing or redistributing any data file.

4. Preparing Bengali text for models

4.1 Why English text tools break on Bengali

Standard NLP recipes are written for English, and applied to Bengali they silently destroy the data:

So this project has its own Bengali text tools (src/text_bn.py).

4.2 What the Bengali text tools do

1. Unicode NFC normalisation — Bengali has several byte-level ways to write the same letter (e.g. য় as one code point vs. য + ়). Everything is normalised so identical text really is identical.
2. Cleaning — regex clean-up that keeps Bengali letters and digits.
3. Tokenising — Bengali-aware word splitting.
4. Stop words — a published Bengali stop-word list, with a protected list (configs/bn_protected_words.txt) that is never removed: the negation words and number words.
5. Stemming — light suffix stripping, and it only cuts if the resulting stem is a real word seen in the training split.

Nine safety rules are checked on the real data (python src/text_bn.py --check), and the module has its own test file. Try it: python src/text_bn.py "কলেজের ছাত্ররা বইটি পড়েনি।" prints every cleaning step.

4.3 Does textbook cleaning help? (ablation)

We tested five cleaning recipes plus one deliberately harmful one:

Two findings:

1. Textbook cleanup is worth almost nothing here. TF-IDF + LogReg moves only 0.542 → 0.551 overall across V0–V4. (Stemming/stop-word removal does buy a smaller model: V4 needs 201,504 features against V0's 259,454, for the same score.)
2. Deleting negation is quietly catastrophic — and invisible in the headline. V2-demo's hard-subset score is 0.609 vs. the default's 0.610 (no visible change) — but its score on contradiction errors collapses from 0.712 to 0.528. The damage only appears when you slice by error type, which is the strongest argument in the project for reporting sliced results. Stop-word removal also actively hurts Skip-gram (0.501 on hard, its worst cell — its best variant is V0, no cleaning at all).

4.4 The three input formats, and what gets cut

Every model gets its input from src/preprocess.py in one of three formats:

$$x = \texttt{[CLS]};\oplus;c;\oplus;\texttt{[SEP]};\oplus;q;\oplus;a;\oplus;\texttt{[SEP]}\qquad(c\text{ omitted when there is no passage})$$

Each part is labelled in Bengali (প্রসঙ্গ: context / প্রশ্ন: question / উত্তর: answer) so the model always knows where each begins.

Models cannot read unlimited text, so input is capped at 256 tokens. Two rules govern what gets cut:

1. Cut the passage — never the question or the answer. Cutting the answer deletes the very thing being judged; cutting the question changes what was asked.
2. Cut from the end — never choose the window by overlap with the answer. That sounds helpful but would leak the label: a correct answer would always find its supporting sentence kept, a made-up one wouldn't, and the model could decide from the passage length alone. It is explicitly forbidden.

In practice this rarely matters: at 256 tokens only 1.5% of passages need cutting at all, and measurable loss of supporting evidence affects 7 of 1,634 checkable pairs (0.4%). (At 128 tokens that would be 7.2% — which is why 256 was chosen.)

5. The models — the ladder, rung by rung

We do not present one model. We present a ladder: each rung adds one capability, so every gain (or lack of one) can be attributed to something specific. Progress is deliberately not assumed to be monotonic — and it isn't.

flowchart LR
    R0["No learning<br/>string rules<br/><b>0.591</b>"] --> R1["Counting words<br/>TF-IDF, Naive Bayes,<br/>LogReg, SVM<br/><b>0.610</b>"]
    R1 --> R2["Word vectors<br/>Skip-gram from<br/>our own text<br/><b>0.529</b>"]
    R2 --> R3["Recurrent nets<br/>RNN, BiRNN, BiLSTM,<br/>+ attention<br/><b>0.605</b>"]
    R3 --> R4["Transformer<br/>trained from scratch<br/><b>0.433</b>"]
    R4 --> R5["Pretrained encoders<br/>BanglaBERT, MuRIL,<br/>XLM-R<br/><b>0.825</b>"]
    style R0 fill:#f3f4f6,stroke:#6b7280,color:#000
    style R4 fill:#fee2e2,stroke:#e34948,color:#000
    style R5 fill:#0ca30c,color:#fff,stroke:none

_{Best hard-subset macro-F₁ per family, dev split.}

Everything below the pretrained rung is trained only on our 6,134 training items, on a laptop CPU, in seconds to minutes. They answer one question: how far can you get using nothing but this data?

5.0 Rung 0 — rules that involve no learning

These are the bars. The fuzzy rule was tuned on train (never dev) and is the number every learned model must beat. Code: src/features.py (python src/features.py --baseline).

5.1 Rung 1 — counting words

Idea in plain words. Count which words appear in an item, then learn which words tend to go with wrong answers. No understanding of order or meaning — a "bag" of words.

• Bag of Words (BoW) uses raw counts; TF-IDF down-weights words that appear everywhere.
• Features: word 1–2-grams plus character 3–5-grams (characters matter in Bengali because inflection and compounding fragment words), one block per record part (passage / question / answer).
• Three learners on top: Naive Bayes, Logistic Regression, linear SVM → six models.
• Code: src/train_classical.py. Cross-checked against a from-scratch Naive Bayes (--reference-check): 100% agreement on 200 dev records.

Result: TF-IDF + LogReg and TF-IDF + SVM tie at 0.610 on hard (McNemar p = 1.000 — see §6.4). One model, BoW + SVM, is flagged unreliable: raw counts run up to 55 with no ceiling and the SVM never converges even after 20,000 iterations; TF-IDF rescales everything to ≤ 1 and settles in seconds. We report its score but mark it — why it fails shows exactly what TF-IDF's weighting is for.

5.2 Rung 1b — a character language model

Idea. Build one model of "how correct answers are written" and one of "how wrong ones are", each predicting the next character from the previous few. Then ask which model finds a new answer more likely.

• Order-4 character n-grams with add-k smoothing (k = 0.1, tuned on train).
• Used two ways: M11a scores an answer against its own passage; M11b is a class-conditional classifier on the answer text alone.

Result: the answer-only character model scores 0.605 on hard (0.529 overall). Its answer-style classifier (M11b) reaches 0.529–0.546 macro-F₁ overall — essentially the same as the answer-only probe (0.542), i.e. there is no generator fingerprint in the answer's writing style that gives the label away.

5.3 Rung 2 — similarity features (and a cautionary tale)

Idea. Compare the answer with its own passage and question using seven hand-made numbers: whether the answer appears verbatim, token overlap, edit distance to the best-matching piece of the passage, how "passage-like" the answer looks to a language model, how much it repeats the question, answer length, and digit share. Feed them to Logistic Regression.

Rule: features are computed from a record and only its own question and passage — never by comparing it with other records. Anything else is retrieval.

Result — and why this model matters more than its score:

This model learned the string-match shortcut, not the task. Judged by the overall number it looks like the project's best result. Judged on the hard subset it's nearly the worst. A single averaged score would have presented it as a triumph; splitting by measured difficulty exposes it immediately.

5.4 Rung 3 — Skip-gram word vectors, written from scratch

Idea. Learn a list of 200 numbers per word by reading text and noticing which words keep similar company — so "similar" words get similar numbers. Average them to represent a whole answer.

• Trained only on our train split (pretrained fastText vectors were deliberately not used, to keep this rung "what our own data gives you"). 17,701 words survive the minimum-count filter; the vectors cover 94% of dev words.
• The usual library (gensim) has no build for Python 3.14, so Skip-gram is implemented from scratch in NumPy (src/skipgram.py), using negative sampling (5 fake neighbours per real one) instead of a full softmax over the vocabulary, which would be far too slow.
• Sanity check (python src/skipgram.py --check): neighbours are sensible — e.g. ১৯৭১ → মার্চ, জিয়াউর; সরকার → মুজিবনগর, প্রবাসী.
• Two ways to summarise an answer (mean, TF-IDF-weighted mean) × two classifiers (LogReg, XGBoost) → four models.

Result: 0.500–0.529 on hard — below simple word counting. Averaging word vectors throws away word order, and 6,134 items is too little text to learn good vectors.

5.5 Rung 4 — recurrent neural networks

Idea. Read the text in order, one word at a time, keeping a running "memory" — so unlike a bag of words these models can see sequence.

flowchart LR
    T["Token ids<br/>passage, question, answer"] --> E["Embeddings 200-d<br/>initialised from our Skip-gram"]
    E --> F["LSTM reading<br/>left to right"]
    E --> B["LSTM reading<br/>right to left"]
    F --> H["Hidden states<br/>one per word"]
    B --> H
    H --> A["Attention:<br/>weight each word<br/>by how much it matters"]
    A --> D["Dropout 0.3"]
    D --> L["Linear layer"]
    L --> S["Sigmoid<br/>P(faithful)"]
    style A fill:#dbeafe,stroke:#2a78d6,color:#000
    style S fill:#0ca30c,color:#fff,stroke:none

Four variants, each adding one capability:

Settings: hidden size 128, dropout 0.3, Adam lr 1e-3, batch 32, up to 20 epochs, early stopping on dev loss (patience 3), gradient clipping at 5; all on CPU. Code: src/train_neural.py.

Result: the simplest model wins — RNN 0.605 on hard (overall 0.560 ± 0.009). Bidirectionality, stacking and attention each cost a little; none beats the best word-counting model (0.610). With so little data, extra capacity has nothing to learn from.

5.6 Rung 5 — a Transformer trained from scratch

Idea. The same architecture as BanglaBERT (self-attention layers), but with no pretraining — random starting weights, trained only on our data. 2 layers, 4 heads, d_model 128, sine/cosine position signal, masked mean pooling (src/train_neural.py).

Result — the weakest and least stable system in the project: 0.433 on hard; overall 0.465 ± 0.072, and it collapsed outright on seed 2024 (0.364). The instability is flagged by the code (collapse_warning()), not hidden.

This is the control experiment. Compare it with BanglaBERT: same family of architecture, differing only in whether it has ever read Bengali before.

5.7 Rung 6 — pretrained encoders

Idea. Start from a Transformer that has already read enormous amounts of text, then fine-tune it on our task. It arrives already knowing Bengali grammar and a great deal about the world.

flowchart BT
    IN["Input: CLS, passage, SEP, question, answer, SEP"] --> EMB["Word + position embeddings"]
    EMB --> ATT["Multi-head self-attention<br/>every word looks at every other word"]
    ATT --> FFN["Feed-forward + LayerNorm"]
    FFN -.->|"repeated 12 times"| ATT
    FFN --> POOL["Take the CLS vector h"]
    POOL --> HEAD["One linear layer"]
    HEAD --> OUT["Sigmoid: P(faithful)"]
    style OUT fill:#0ca30c,color:#fff,stroke:none

Each layer applies self-attention, letting every word decide which other words matter:

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)V$$

That is exactly the ability this task needs — to connect "১৯৬৮" to the denomination rather than to the Edinburgh congregation. The classification head is a single linear layer over the [CLS] summary vector, trained with binary cross-entropy:

class FaithfulnessClassifier(nn.Module):
    """Pretrained encoder + one linear layer -> a single score."""
    def __init__(self, encoder, hidden_size, dropout=0.1):
        super().__init__()
        self.encoder = encoder
        self.dropout = nn.Dropout(dropout)
        self.head = nn.Linear(hidden_size, 1)

    def forward(self, input_ids, attention_mask):
        out = self.encoder(input_ids=input_ids, attention_mask=attention_mask)
        cls = out.last_hidden_state[:, 0]          # the [CLS] summary vector
        return self.head(self.dropout(cls)).squeeze(-1)

loss_fn = nn.BCEWithLogitsLoss()   # sigmoid applied inside: more numerically stable

Three encoders compared — they differ mainly in how much of their pretraining was Bengali:

More languages did not help; a larger share of Bengali did. (BERT-style code for mBERT is also included but was not part of the reported runs.)

A detail that is easy to get wrong. BanglaBERT was pretrained on text passed through a specific Bengali normaliser. Feeding it raw text costs accuracy silently — nothing crashes, the score is just lower. This code therefore refuses to run BanglaBERT unless the normaliser is installed: pip install git+https://github.com/csebuetnlp/normalizer.

Code: src/train_transformer.py; GPU driver: notebooks/02_bert_gpu.ipynb (Colab + Drive).

5.8 A large language model as a reference point

For context, a 31-billion-parameter instruction-tuned model (Gemma-4-31B) was run through an external pipeline (a Kaggle notebook: notebooks/final-notebook-bhibranti.ipynb; outputs in results/gemma_reference/). It reaches 0.908 on the hard dev subset, above BanglaBERT's 0.825. This is a reference point, not a comparison, for three reasons:

On the 160 hard dev items its retriever could not match, it scores 0.894 — the more meaningful figure, and still not directly comparable. The comparison shows what a very large model with outside access can reach on this data. It does not show our approach is worse under equal conditions, because the conditions are not equal.

6. How we score things (and why you can trust the numbers)

Every system — from the two-line string rule to BanglaBERT — is scored by one shared file (src/evaluate.py) that was written before the first model, precisely so that no model can be judged on its own terms.

6.1 The metric: macro-F1 explained

Macro-F₁ computes an F₁ score (a blend of precision and recall) for each class separately, then averages the two classes equally:

$$P_k=\frac{TP_k}{TP_k+FP_k},\quad R_k=\frac{TP_k}{TP_k+FN_k},\quad F_1^{(k)}=\frac{2P_kR_k}{P_k+R_k},\quad \text{macro-}F_1=\tfrac12\big(F_1^{(0)}+F_1^{(1)}\big)$$

Why not plain accuracy? Macro-averaging refuses to reward laziness. Take a "system" that ignores its input and always answers faithful. On our perfectly balanced corpus:

Plain accuracy would flatter it at 0.50 even though it never catches a single hallucination. 0.333 is the floor every system must clear before it has demonstrated anything. (BLEU/ROUGE are absent on purpose: they measure text overlap for generation tasks; ours makes one binary decision per item.)

6.2 Everything is sliced

A single blended number is never reported alone. Every model is broken down by:

The string baselines are recomputed on exactly the records being scored, never copied from another table, and slices under 60 records are marked "small".

6.3 Fairness commitments

1. One scorer for everything.
2. Three seeds (42, 1337, 2024) for anything with randomness; mean ± standard deviation reported.
3. The test split is opened once. All development used dev. (The demo and notebooks never touch test.)
4. Every run is logged — including failures — to results/experiment_log.csv, the single source of truth. The demo's scoreboard is read live from it, so it can never drift from the paper.
5. Any macro-F₁ above 0.95 is treated as leakage until proven otherwise.

6.4 Is one model really better than another?

Two scores that differ by 0.01 may just be noise. We use McNemar's test, which looks only at the items where the two models disagree and asks whether one wins those disagreements far more often than chance (50/50).

Confidence intervals are computed by resampling pairs, not items — the two answers to one question share a passage and are not independent observations, so treating them as independent would give an interval that is too narrow and an unjustified claim of significance.

7. Results

All numbers: macro-F₁ on the development split. "Hard" = items with a passage where string matching fails.

7.1 The full ladder

7.2 Findings

1. Only BanglaBERT clearly beats the rules — and it isn't luck. 0.825 vs. 0.591 on hard; McNemar p < 0.001 (see §6.4).

2. Without pretraining, nothing gets close. The best system that never saw outside text reaches 0.610 — barely above the 0.591 rule. Counting words, custom word vectors and recurrent networks all land in the same narrow 0.49–0.61 band. With 6,134 training items that is the expected outcome, not an implementation failure: there simply isn't enough data to learn Bengali and the task at once.

3. Monolingual pretraining beats multilingual pretraining. BanglaBERT (0.825) > MuRIL (0.771) > XLM-R (0.623).

4. What pretraining is worth. The from-scratch Transformer (0.433 hard; 0.465 overall) and BanglaBERT (0.825; 0.820) are the same kind of network. The +0.355 overall gap is a direct measurement of what pretraining contributes on this task.

5. A model that looks good and isn't. Similarity features: best overall (0.730), near-worst hard (0.490). Never report an unsplit score.

7.3 By subject

The pattern is consistent and explainable. Reading comprehension scores 0.963 because the evidence is in the passage — the model only has to check it. Civil-service (BCS) questions score 0.517, barely above guessing, because answering them requires knowing a fact that appears nowhere in the input. A closed-book model cannot verify what it was never told. We report this rather than hide it, because it shows precisely where the approach reaches its limit.

7.4 By kind of mistake

Contradictions and wrong numbers are caught almost perfectly — both leave a visible trace in the text (a negation word; a digit that doesn't match). Fabrication is caught least often (0.711) and is also the most common error type (42.3% of all wrong answers) — the largest remaining source of error, and the one category that cannot be resolved by comparing against a passage, because a fabricated answer has no counterpart there to compare against.

7.5 Where BanglaBERT still fails

• Of the 619 hallucinated dev answers, it catches 520 and misses 99, while wrongly flagging 124 correct answers.
• With a passage: 0.925. Without one: 0.680. Two-thirds of the remaining error is in items that must be judged from knowledge alone.

8. The interactive demo

python src/serving.py --build     # once: fits every model and caches it (CPU only; the 12 classical
                                  # models take ~6 min, the neural models a few more)
python src/app.py                 # then open http://127.0.0.1:8000

Check an answer

Type (or click) a question, an answer and an optional passage. Every model votes, and the page shows the evidence behind each vote next to the two no-learning rules — because a model that agrees with a string matcher has not demonstrated anything. Models are listed by past performance on hard questions, not by confidence today.

• Examples marked BERT ✓ are pairs BanglaBERT gets right on both answers; click the green (true) or red (made-up) answer to prefill the form.
• For real dev examples, a Gemma-4-31B reference verdict appears (read from its cached results — it is not run live, and it is clearly labelled as not comparable).
• Note how the models disagree on a hard item. That is the finding, not a bug.

Browse data

Page through real train and dev records — context, question, answer, label 1/0 — as plain selectable text to copy into the checker, or click check it to prefill it. (The test split is deliberately not browsable.)

Model scores

The full scoreboard, ranked by hard-subset score, with the baselines to beat and plain-language notes on how to read it. Numbers are read live from results/experiment_log.csv.

What is and isn't served. The demo loads 12 classical + 5 neural models, plus BanglaBERT if its weights are in models/banglabert_s42/ and the normaliser is installed. Weights are large and git-ignored: produce them with notebooks/02_bert_gpu.ipynb on a GPU. MuRIL and XLM-R scores appear in the experiment log and paper but are not wired into the demo.

9. Reproduce it yourself

9.1 Requirements

• Python 3.14 (verified on 3.14.2, Windows 11). Dependencies are pinned in requirements.txt.
• CPU is enough for everything except fine-tuning the pretrained encoders (that wants a GPU — a free Colab T4 works).
• Data and cached models take a few hundred MB; the BanglaBERT weights add roughly 440 MB.

python -m venv .venv && source .venv/Scripts/activate      # Windows / Git Bash
pip install -r requirements.txt
# Only for BanglaBERT (it refuses to run without this):
pip install git+https://github.com/csebuetnlp/normalizer

9.2 The five-minute path

python src/audit.py --data data/splits --probe metadata    # 1. the data gate passes (0.514 < 0.60)
python src/evaluate.py --baselines --table                 # 2. the no-learning rules reproduce
python src/train_classical.py --model tfidf_logreg         # 3. one real model, full report
python src/serving.py --build && python src/app.py         # 4. the demo

Or open notebooks/00_full_walkthrough.ipynb — the whole pipeline end to end with charts (~9 minutes) — or notebooks/INFERENCE.ipynb.

9.3 Full pipeline

# --- Data (deterministic, seed 42) --------------------------------------------------
python src/build_bn_pool.py          # raw -> data/interim/bn_pool.jsonl
python src/build_corpus.py           # interim -> data/corpus/ + data/splits/
python src/merge_annotation.py       # RE-RUN after any corpus rebuild: restores `excluded` + error types
python src/audit.py --data data/splits --probe metadata      # THE GATE (exits non-zero if it fails)

# --- Annotation checks ---------------------------------------------------------------
python src/score_agreement.py --a <sheetA.csv> --b <sheetB.csv>                        # pilot kappa
python src/score_test_agreement.py --a-dir data/annotated/round1/test/tawhid \
                                   --b-dir data/annotated/round1/test/shejan          # kappa = 0.865

# --- Text tools & features -------------------------------------------------------------
python src/text_bn.py --check        # 9 safety rules on the real data
python src/features.py --check       # feature sanity rules
python src/features.py --baseline    # tune + score the fuzzy rule
python src/preprocess.py --check     # input-format guarantees

# --- Models ------------------------------------------------------------------------------
python src/skipgram.py --train                       # ~3 min: word vectors from our own text
python src/train_classical.py --all --seeds --log    # every classical model, 3 seeds, recorded
python src/train_classical.py --ablation             # the preprocessing experiment
python src/train_neural.py --all --seeds --log       # RNN family + Transformer, 3 seeds
python src/train_neural.py --check                   # prove the three fixes to naive implementations
python src/train_transformer.py --model csebuetnlp/banglabert --seed 42   # GPU

# --- Scoring -----------------------------------------------------------------------------
python src/evaluate.py --baselines --table           # rule baselines on dev -> results/tables/
python src/evaluate.py --demo                        # what the numbers mean, on made-up models
#   Every trained model is scored through evaluate.py inside its own training script
#   (add --log to record the run in results/experiment_log.csv).
#   The test split is opened EXACTLY ONCE, for the final system, and is guarded:
#   --split test refuses to run without --final.

# --- Demo --------------------------------------------------------------------------------
python src/serving.py --build && python src/app.py

9.4 Tests

python -m pytest tests/ -q

Roughly 280 tests, one file per src/ module, each written to fail loudly if a documented rule is broken (the neural-model tests are mutation-tested: removing any of the three fixes makes a named test fail). The tests need no GPU and no downloads. Known issue: on Windows, one subprocess test in test_train_transformer.py currently fails with a console-encoding (cp1252) decoding error in the test harness itself; it is unrelated to the code under test and is present on the untouched code as well.

9.5 Reproducibility notes

• All randomness is seeded (42, 1337, 2024). Reruns reproduce exactly on the same hardware and library versions.
• results/experiment_log.csv records every run: model, input format, preprocessing, seed, learning rate, batch, epochs, dev/test score, and notes.
• The pretrained-encoder runs used Colab GPUs; tiny numerical differences on other hardware are normal.

10. Repository guide

10.1 Layout

Bhibranti/
├── src/            The pipeline: data building, text tools, features, every model, scoring, demo server
├── tests/          One test file per src/ module
├── notebooks/      Runnable walkthroughs (full pipeline, GPU training drivers, inference demo, LLM reference)
├── web/            Templates + stylesheet for the demo (served by src/app.py)
├── data/           Corpus, splits, annotation records, source register (load ONLY via src/splits.py)
├── configs/        Frozen record schema + Bengali word lists (stop words, protected words, suffixes)
├── results/        experiment_log.csv (every run), scored tables, the external Gemma reference run
├── docs/           Requirements, implementation guide, annotation guidelines, the paper, README charts
├── models/         (git-ignored) fine-tuned checkpoints
└── requirements.txt, LICENSE

10.2 Source modules (src/)

10.3 Notebooks

Notebooks are deliberately thin: they import the tested modules from src/ rather than re-implementing them, so a notebook can never drift from the code the numbers came from.

10.4 Working with the data

from splits import load_split          # run from src/, or add src/ to sys.path
train = load_split("train")            # excluded pairs already removed
dev   = load_split("dev")
# Do NOT load the test split until the single sanctioned final evaluation.

Reading the .jsonl files directly would silently train and score on the 180 pairs that humans found unusable.

10.5 Documentation

11. FAQ and troubleshooting

12. Limitations, ethics and future work

Limitations

• Scope. One language, one domain, short factual answers. Not shown to transfer to conversational Bengali, long-form answers or other languages.
• The pairs are LLM-constructed. Labels are human-verified, but the wrong answers were written with LLM assistance from source text (see §3.1), so they may not resemble hallucinations from a particular real system. Treat results as measuring detection on this distribution.
• Scale. ~6,000 training items is small next to English benchmarks. The from-scratch Transformer varies by ±0.072 across seeds and collapsed on one.
• Development scores are mildly optimistic. The dev split chose the stopping epoch and best checkpoint. The test split — opened once — is the honest estimate.
• Some questions can't be answered this way at all. With no passage and an outside-knowledge answer, a closed-book model cannot verify anything. The 0.517 civil-service score is that limit made visible.
• Labels, not explanations. The system outputs a binary decision, not which span of the answer is wrong.
• The ladder is incomplete. mBERT and IndicBERT v2, further pretraining of multilingual encoders, and a soft-voting ensemble were proposed but not completed; the claim that pretraining is decisive rests on three encoders.

Ethics and responsible use

• This is a support tool, not a grading authority. It is wrong on roughly one hard item in six, and substantially weaker without a passage (0.680). It must not be used to mark student work automatically or to reject answers without human review. Its per-subject spread (0.963 → 0.517) means a single overall figure would badly misrepresent its reliability on any particular topic.
• Data. Wikipedia-derived text (CC BY-SA) and exam question banks; factual and encyclopaedic, no personal information about private individuals. Licence status is documented — and partly unresolved — in data/SOURCES.md.
• Annotation. Carried out by the authors, both fluent Bengali speakers; no external annotators, so no compensation questions arise. The material is academic and not distressing.
• Environmental cost. Fine-tuning took ≈ 40 minutes per run on one T4; the whole set of experiments used a few tens of GPU-hours.

Future work (in the order we judge most likely to help)

1. Further pretraining of a multilingual encoder on Bengali text — a direct test of whether BanglaBERT's edge over MuRIL comes from the amount of Bengali seen or from the architecture.
2. A different mechanism for fabrication — the most common error (42.3%) and least often caught (0.711), which cannot be checked against a passage.
3. The remaining encoders (mBERT, IndicBERT v2) to test whether the monolingual advantage holds more widely.
4. A soft-voting ensemble and an input-format comparison — cheap once the encoders exist.
5. Phase 2: Banglish / romanised Bengali and code-mixed text, and span-level (which-part-is-wrong) detection — planned, deliberately not started.

13. Citation and licence

@misc{banghallu2026,
  title  = {BangHallu: A Bengali Corpus for Hallucination Detection
            in Large Language Models},
  author = {Hasan, Tawhidul and Shejan, Md. Saif Ahmed},
  year   = {2026},
  note   = {Final Project Report, Khulna University of Engineering \& Technology}
}

Code is released under the MIT licence. Data under data/ is not covered by that licence: it is derived from Bengali Wikipedia (CC BY-SA 4.0 — attribution and share-alike apply) and from question banks whose redistribution terms are still being confirmed. See data/SOURCES.md before reusing any data file.

Related work we build on and compare against: BenHalluEval (Bengali generation-side hallucination), Poly-FEVER (multilingual fact verification), BanglaQuAD and BEnQA (Bengali QA resources), and the encoders BanglaBERT, MuRIL and XLM-R. Full references are in docs/paper/main.tex.

Authors: Tawhidul Hasan and Md. Saif Ahmed Shejan, Department of Computer Science and Engineering, Khulna University of Engineering & Technology (KUET). Supervised by Dr. K. M. Azharul Hasan and Md. Nazirulhasan Shawon.