ESGenius

Benchmarking LLMs on Environmental, Social, and Governance (ESG) and Sustainability Knowledge

EMNLP 2025 Main Conference Oral | Resource and Theme Paper Award nominations, Top 1%

Project Site | Interactive Heatmap | Hugging Face Dataset | ACL Anthology | PDF | Dataset Card | Evaluation Guide

---

ESGenius is an expert multiple-choice benchmark for evaluating whether large language models understand ESG and sustainability knowledge in standards-aware settings. It includes 1,136 source-grounded questions, plain and reference-aware dataset files, evaluation scripts, published result figures, and an interactive 50-model heatmap for question-level diagnosis.

Quick Links

At a Glance

Why ESGenius?

General factual benchmarks do not fully capture the domain-specific demands of ESG work. Sustainability reporting, climate disclosure, and governance analysis require specialized terminology, source-dependent concepts, and knowledge of reporting frameworks. ESGenius targets this gap directly.

• Covers environmental, social, governance, and sustainability knowledge across major standards and disclosure contexts.
• Draws from IPCC, GRI, SASB, ISO, IFRS/ISSB, TCFD, CDP, and related ESG sources.
• Uses a controlled A-D multiple-choice format with Z for abstention or uncertainty.
• Provides source-grounded reference fields for audit, retrieval, and citation-aware evaluation.
• Includes reproducible evaluation scripts and published result artifacts.

What Is Included

Repository Layout

Hugging Face Release

The canonical public benchmark bundle is hosted at cy0307/ESGenius. It contains the dataset files, dataset documentation, citation metadata, license, and lightweight evaluation scripts so users can download the benchmark from the Hugging Face Hub while using GitHub for source development and the project site.

hf download cy0307/ESGenius \
  --type dataset \
  --local-dir ESGenius-HF

Dataset

The public dataset lives in data/ and is mirrored on Hugging Face. Use the plain files for standard model evaluation and the reference-aware file for audit or retrieval experiments.

Core fields:

See data/README.md for schema notes and usage guidance.

Quick Start

Create an environment:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Copy the environment template:

cp .env.example .env

Run a small local smoke test:

python eval_opensource.py \
  --dataset ESGenius_1136q.csv \
  --models Qwen/Qwen2.5-0.5B-Instruct \
  --limit 10

Results are written to results/ as Excel workbooks with summary and details sheets.

Evaluation Paths

All evaluation paths share normalization, option extraction, metric computation, and workbook export through evaluation_utils.py.

Reference-aware smoke test:

python eval_opensource_rag.py \
  --dataset ESGenius_w_ref_1136q.csv \
  --models Qwen/Qwen2.5-0.5B-Instruct \
  --limit 10

Qwen API smoke test:

python eval_qwen_api.py \
  --dataset ESGenius_1136q.csv \
  --models Qwen2.5-Max \
  --limit 10

For all options, output structure, and reproducibility notes, see docs/evaluation.md.

Results and Project Site

The project site follows a paper-project format and keeps the homepage lightweight. The full diagnostic view lives in the interactive heatmap.

• Project homepage
• Interactive heatmap
• Hugging Face dataset
• ACL Anthology record

Main ESGenius benchmark results. Additional figures are available in figures/, results/, and on the project site.

Validate the static site locally:

python scripts/check_static_site.py
python -m http.server 8000

Then open http://127.0.0.1:8000/.

Reproducibility Checklist

• Keep query_id as a string when loading datasets.
• Keep the Z option in prompts; it is part of the benchmark protocol.
• Use deterministic generation settings from evaluation_utils.py.
• Existing result workbooks are skipped by default to avoid accidental overwrites.
• Use --force only when intentionally regenerating outputs.
• Report both accuracy and invalid or abstention behavior when comparing models.
• Use the reference-aware CSV when an experiment depends on source snippets.

Citation

If you use ESGenius, please cite the EMNLP 2025 paper and repository metadata in CITATION.cff.

@inproceedings{he-etal-2025-esgenius,
  title = "{ESG}enius: Benchmarking {LLM}s on Environmental, Social, and Governance ({ESG}) and Sustainability Knowledge",
  author = "He, Chaoyue and Zhou, Xin and Wu, Yi and Yu, Xinjia and Zhang, Yan and Zhang, Lei and Wang, Di and Lyu, Shengfei and Xu, Hong and Xiaoqiao, Wang and Liu, Wei and Miao, Chunyan",
  editor = "Christodoulopoulos, Christos and Chakraborty, Tanmoy and Rose, Carolyn and Peng, Violet",
  booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
  month = nov,
  year = "2025",
  address = "Suzhou, China",
  publisher = "Association for Computational Linguistics",
  url = "https://aclanthology.org/2025.emnlp-main.739/",
  doi = "10.18653/v1/2025.emnlp-main.739",
  pages = "14612--14653",
  ISBN = "979-8-89176-332-6"
}

Contributing

Please see CONTRIBUTING.md for contribution guidance. For vulnerability reporting, see SECURITY.md.

License

This project is released under the Apache 2.0 License.