nlp-summarization-toolkit

Inference-only summarization for English and Bengali using Hugging Face checkpoints (BART, T5, PEGASUS, mT5, BanglaT5). Includes Bengali text helpers, optional XL-Sum loader, and metrics (ROUGE, BERTScore, n-gram diversity).

Install

python -m venv .venv
.venv\Scripts\activate
pip install -e .

BanglaT5 expects the csebuetnlp normalizer:

pip install git+https://github.com/csebuetnlp/normalizer

Python

from toolkit.model.bart_model import BartModel
from toolkit.model.multilingual.banglat5_model import BanglaT5Model

en = BartModel()
en.summarize(["Long English article text..."])

bn = BanglaT5Model(model_size="small")
bn.summarize(["বাংলা টেক্সট..."])

CLI

python scripts/run_summarization.py --model bart --language en --input_file docs.txt --output_file out.txt
python scripts/run_summarization.py --model banglat5 --language bn --preprocessing --evaluate --input_file in.txt --output_file out.txt --reference_file refs.txt

After install: summ-run with the same flags.

Warm caches (models are large):

python scripts/warm_up_models.py
python scripts/warm_up_models.py --models all

Models

Tests

python -m unittest tests.test_bn_utils tests.test_evaluation tests.test_models -v

Runs offline; tests that need downloaded weights are skipped.

Layout

• scripts/ — run_summarization.py, warm_up_models.py
• toolkit/model/ — model wrappers
• toolkit/evaluation/ — metrics
• toolkit/util/bn_text_utils.py — Bengali cleaning helpers
• toolkit/dataset/xlsum_bengali.py — XL-Sum Bengali loader