CB-LLM for Scientific Claim-Type Classification

An interpretable-by-design extension of CB-LLM (ICLR 2025) that classifies each sentence of a research-paper abstract into its rhetorical role — background, hypothesis, method, result, or conclusion — and shows which human-readable concepts drove every prediction.

[S4] Pain scores decreased by 30% in the exercise group versus 8% in controls (p<0.001).
→ Experimental / Empirical claim (RESULTS)
  driven by:
  - reports a p-value (0.812)
  - states a statistically significant difference (0.604)
  - reports a numerical outcome of the study (0.551)

Why interpretability, not just accuracy

Most claim-classification work reports accuracy and stops there — a black-box model tells you what it decided but not why. CB-LLM inserts a concept bottleneck layer between the language model and the final classifier: predictions are forced to route through a small set of human-readable concepts (e.g. "reports a p-value", "states a causal claim"), so every classification comes with a built-in explanation rather than a post-hoc guess. This extension keeps that property while moving CB-LLM from short single-sentence datasets (SST2, AG News) to multi-sentence scientific abstracts, which is a meaningfully harder setting: sentence roles depend on surrounding context, and the dataset is naturally class-imbalanced.

What it does

Takes the PubMed 20k RCT dataset — abstracts from randomized controlled trials, each sentence labeled by its structural role — and reframes those five roles as claim types:

A concept bank of 200 concepts (40 per class) drives the bottleneck layer. Input: one abstract. Output: every sentence tagged with a claim type plus the specific concepts that explain the call.

Note: RCT section-heading labels are a reasonable but imperfect proxy for "true" claim type — worth stating explicitly in any write-up using this.

Pipeline

Abstract → sentence split → RoBERTa backbone → Concept Bottleneck Layer (200 concepts,
                                                  automatic concept correction) → Linear
                                                  predictor → claim type + concept attribution

Three-step training, same shape as the original CB-LLM pipeline:

1. Automatic Concept Scoring — score every sentence against the concept bank
2. Train the Concept Bottleneck Layer — with Automatic Concept Correction (ACC)
3. Train the final linear predictor — maps concept activations → claim type

A dataset-subsetting utility (pubmed_subset.py) makes iteration fast: train/evaluate on a reproducible 25%/50%/100% slice by abstract (never splitting a sentence from its abstract), with cached subsets so repeated runs don't regenerate data.

Results

Evaluated with macro-F1, not plain accuracy — PubMed-RCT is class-imbalanced (RESULTS/METHODS dominate), so accuracy alone would be misleading. test_CBLLM.py reports accuracy, macro-F1, and a full per-class precision/recall/F1 table, with a black-box (no bottleneck) baseline available for comparison via finetune_black_box.py.

Tech stack

Python · PyTorch · RoBERTa (via the CB-LLM backbone, GPT-2 also supported) · scikit-learn · CUDA 12.1 recommended

Quick start

# 1. Get CB-LLM's classification/ folder, then copy every file from this repo into it
#    (concepts.py, config.py, train_CBL.py, etc. — all are drop-in replacements or additions)
cd classification
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt && pip install scikit-learn

# 2. Get the dataset
git clone https://github.com/Franck-Dernoncourt/pubmed-rct.git
python pubmed_loader.py   # sanity check: prints split sizes + label distribution

# 3. Train (same 3 steps as the original SST2/AG News pipeline)
python get_concept_labels.py --dataset pubmed_rct
python train_CBL.py --dataset pubmed_rct --automatic_concept_correction
python train_FL.py --cbl_path mpnet_acs/pubmed_rct/roberta_cbm/cbl_acc.pt

# 4. Evaluate + see it work on a real abstract
python test_CBLLM.py --cbl_path mpnet_acs/pubmed_rct/roberta_cbm/cbl_acc.pt --sparse
python demo_classify_abstract.py --cbl_path mpnet_acs/pubmed_rct/roberta_cbm/cbl_acc.pt --sparse

python print_concept_activations.py --cbl_path <path>
python print_concept_contributions.py --cbl_path <path>

Disclaimer

Built as a research-methods project extending Trustworthy-ML-Lab/CB-LLMs (ICLR 2025) to a new dataset and task framing. Not affiliated with the original authors.