MULTILINGUAL-LLM-SUMMARIZATION

This repository provides a comprehensive, reproducible zero-shot evaluation framework for Large Language Models (LLMs) on abstractive news summarization across Bengali and English using the XL-Sum benchmark dataset (1,012 test items each).

---

Project Overview & Methodology

The goal of this benchmark is to establish performance baselines across closed-source frontier models (OpenAI GPT series, Google Gemini) and open-weights models (Meta LLaMA, Alibaba Qwen, DeepSeek).

Key Evaluation Features:

• Linguistic & Statistical Metrics: Evaluates ROUGE-1, ROUGE-2, ROUGE-L, BLEU, chrF, and BERTScore-Recall.
• Recommended Metric Selection (based on SummEval & BanglaSummEval literature):
  • ROUGE-1 / ROUGE-2: Prioritized over ROUGE-L to reward semantic quality without penalizing structural restructuring.
  • chrF: Captures character n-gram overlap, showing strong human correlation (r = 0.588).
  • BERTScore-Recall: Assesses factual coverage using roberta-base for English and xlm-roberta-base for Bengali.
• LLM-as-a-Judge: Incorporates G-Eval multi-aspect evaluation (Fluency, Coherence, Relevance, Consistency).
• Auto-Resume Pipeline: API evaluation runners incrementally save outputs per sample, avoiding duplicate execution or API quota wastage.

---

Benchmark Results (1,012 Test Items)

1. English Abstractive Summarization Benchmark

---

2. Bengali Abstractive Summarization Benchmark

---

3. Statistical Significance & Confidence Intervals

To ensure rigorous evaluation, standard deviations (SD), 95% bootstrap confidence intervals (CI), and paired statistical significance tests (paired $t$-statistic, asymptotic $p$-value, and bootstrap $p$-value with 5,000 paired resamples) are computed consistently across all reported comparisons:

Unified Statistical Significance & Comparison Table

Comparison-by-Comparison Detail

• Bengali Factuality (BanglaSummEval BSE-F1):

  • Gemini 2.5 Flash: Mean = 58.43%, SD = 16.77, 95% CI: [55.08, 61.60] ($N = 100$)
  • LLaMA 3.3 70B: Mean = 58.28%, SD = 16.64, 95% CI: [54.92, 61.35] ($N = 100$)
  • Mean Difference: +0.15% (unrounded: +0.146%)
  • Paired Test Statistic: $t = 0.098$ ($N = 100$)
  • Paired-test $p$-value: $p = 0.9218$
  • Bootstrap $p$-value (5,000 resamples): $p = 0.9162$
  • Finding: The difference is not statistically significant.

• English Factuality (SummEval SE-F1):

  • DeepSeek V4 Flash: Mean = 63.45%, SD = 15.16, 95% CI: [60.53, 66.48] ($N = 100$)
  • LLaMA 3.3 70B: Mean = 60.18%, SD = 17.11, 95% CI: [56.84, 63.54] ($N = 100$)
  • Mean Difference: +3.27% (unrounded: +3.271%)
  • Paired Test Statistic: $t = 2.407$ ($N = 100$)
  • Paired-test $p$-value: $p = 0.0161$
  • Bootstrap $p$-value (5,000 resamples): $p = 0.0130$
  • Finding: Statistically significant ($p < 0.05$).

• Bengali ROUGE-1 (LLaMA 3.3 70B vs. GPT-4o Mini):

  • LLaMA 3.3 70B: Mean = 16.23%, SD = 9.05, 95% CI: [15.69, 16.79] ($N = 1012$)
  • GPT-4o Mini: Mean = 15.63%, SD = 8.27, 95% CI: [15.12, 16.13] ($N = 1012$)
  • Mean Difference: +0.61% (unrounded: +0.608%)
  • Paired Test Statistic: $t = 2.713$ ($N = 1012$)
  • Paired-test $p$-value: $p = 0.0067$
  • Bootstrap $p$-value (5,000 resamples): $p = 0.0070$
  • Finding: Statistically significant ($p < 0.01$).

• Bengali ROUGE-1 (LLaMA 3.3 70B vs. Gemini 2.5 Flash):

  • LLaMA 3.3 70B: Mean = 16.23%, SD = 9.05, 95% CI: [15.69, 16.79] ($N = 1012$)
  • Gemini 2.5 Flash: Mean = 15.05%, SD = 7.86, 95% CI: [14.55, 15.52] ($N = 1012$)
  • Mean Difference: +1.19% (unrounded: +1.187%)
  • Paired Test Statistic: $t = 5.364$ ($N = 1012$)
  • Paired-test $p$-value: $p < 0.0001$ ($p = 8.12 \times 10^{-8}$)
  • Bootstrap $p$-value (5,000 resamples): $p < 0.0001$ (bootstrap $p = 0.0000$)
  • Finding: Statistically significant ($p < 0.0001$).

• English ROUGE-1 (GPT-5.6 Luna vs. Gemini 3.1 Flash Lite):

  • GPT-5.6 Luna: Mean = 24.12%, SD = 7.61, 95% CI: [23.65, 24.59] ($N = 1012$)
  • Gemini 3.1 Flash Lite: Mean = 22.94%, SD = 6.60, 95% CI: [22.54, 23.35] ($N = 1011$)
  • Mean Difference: +1.17% (unrounded: +1.173%)
  • Paired Test Statistic: $t = 6.321$ ($N = 1011$)
  • Paired-test $p$-value: $p < 0.0001$ ($p = 2.60 \times 10^{-10}$)
  • Bootstrap $p$-value (5,000 resamples): $p < 0.0001$ (bootstrap $p = 0.0000$)
  • Finding: Statistically significant ($p < 0.0001$).

To run the full statistical significance suite across all models and metrics:

python src/evaluation/compute_stats.py

---

Repository Organization

multilingual-llm-summarization/
├── README.md                           # Main publication landing page & benchmark leaderboard
├── LICENSE                             # MIT License
├── requirements.txt                    # Pinned python dependencies
├── .gitignore                          # Excludes credentials, caches, datasets, TeX files
│
├── data/                               # Dataset documentation & schemas
│   └── README.md                       # Instructions to download XL-Sum Bengali/English datasets
│
├── src/                                # Source code
│   ├── evaluation/                     # Metric engines (ROUGE, chrF, BLEU, BERTScore, LLM Judge)
│   │   ├── benchmark_7_models.py
│   │   ├── compute_bn_bertscore.py
│   │   ├── evaluate_all_16_files_judge.py
│   │   └── llm_judge_evaluate.py
│   ├── models/                         # API Providers & Evaluation Wrappers
│   │   ├── evaluate_openai_models.py
│   │   ├── evaluate_gemini_models.py
│   │   ├── evaluate_openrouter_models.py
│   │   └── evaluate_groq_qwen.py
│   └── visualization/                  # Publication visualizers
│       └── generate_paper_plots.py     # Generates radar, factuality, and ROUGE plots
│
├── results/                            # Benchmark Output Datasets
│   ├── bengali/                        # Bengali output CSV files per model
│   └── english/                        # English output CSV files per model
│
├── figures/                            # Publication-ready figures (.pdf and .png)
│
└── tools/                              # Data inspection & diagnostic scripts
    ├── check_anomalies.py
    └── extract_error_samples.py

---

Getting Started

1. Installation

Clone the repository and install required packages:

git clone https://github.com/khanmahijyoti/multilingual-llm-summarization.git
cd multilingual-llm-summarization
pip install -r requirements.txt

2. Environment Variables

Set your API keys depending on the models you wish to evaluate:

export OPENAI_API_KEY="your-openai-key"
export GEMINI_API_KEY="your-gemini-key"
export OPENROUTER_API_KEY="your-openrouter-key"
export GROQ_API_KEY="your-groq-key"

3. Running Metric Evaluation

To compute all metrics across generated model summaries:

python src/evaluation/benchmark_7_models.py

To generate publication figures:

python src/visualization/generate_paper_plots.py