ChemBench

---

⚠️ IMPORTANT NOTICE - NOT FOR TRAINING

---

📋 Dataset Summary

ChemBench is a meticulously crafted benchmark designed to assess the chemistry and materials science capabilities of Large Language Models (LLMs). 🧪 This comprehensive evaluation suite spans diverse chemical disciplines and complexity levels, from straightforward multiple-choice questions to sophisticated open-ended reasoning challenges that demand both deep chemical knowledge and advanced reasoning skills.

The benchmark comprises over 2,700 high-quality questions manually curated by chemistry and materials science experts. Each question is designed to test specific aspects of chemical understanding, making ChemBench an invaluable resource for researchers developing and evaluating AI systems in the chemical sciences. 🔬

📊 Dataset Statistics

• 🎯 Total Questions: 2,700+ expertly curated questions
• 👨‍🔬 Expert Curation: Manually created by chemistry and materials science professionals
• 🌐 Multi-field Coverage: Spanning 9 major chemistry and materials science domains
• 📈 Complexity Range: From basic concepts to advanced reasoning challenges
• ⚖️ Question Types: Multiple-choice and open-ended format questions

🗂️ Dataset Configurations

ChemBench encompasses nine major chemistry and materials science domains, each designed to evaluate specific aspects of chemical knowledge and reasoning:

🔬 Analytical Chemistry (analytical_chemistry)

Questions covering spectroscopic techniques, chromatography, mass spectrometry, and other analytical methods essential for chemical analysis and characterization.

🧬 Chemical Preference (chemical_preference)

Human preference-based questions evaluating compounds on oral bioavailability, toxicity profiles, drug-likeness, and other pharmacologically relevant properties.

⚗️ General Chemistry (general_chemistry)

Fundamental chemistry concepts including periodic table properties, chemical bonding theories, stoichiometry, and basic thermodynamics.

🔵 Inorganic Chemistry (inorganic_chemistry)

Properties and reactions of inorganic compounds, coordination chemistry, organometallics, solid-state chemistry, and crystal structures.

�️ Materials Science (materials_science)

Material properties and applications covering polymers, ceramics, nanomaterials, composites, and advanced functional materials.

🌿 Organic Chemistry (organic_chemistry)

Organic compound properties, reaction mechanisms, functional group chemistry, synthesis strategies, and stereochemistry.

⚡ Physical Chemistry (physical_chemistry)

Fundamental principles including thermodynamics, kinetics, electrochemistry, quantum chemistry, and statistical mechanics.

⚙️ Technical Chemistry (technical_chemistry)

Practical applications in chemical engineering, process design, industrial chemistry, and chemical manufacturing.

⚠️ Toxicity and Safety (toxicity_and_safety)

Chemical safety assessment, environmental chemistry, toxicology, risk assessment, and regulatory compliance.

📜 License

All content is made open-source under the MIT license, allowing for:

• ✅ Commercial and non-commercial use
• ✅ Modification and derivatives
• ✅ Distribution and private use
• ⚠️ Attribution required
• ⚠️ No warranty provided

📖 Data Fields

The dataset contains comprehensive metadata and content fields designed for robust evaluation and analysis:

🔍 Core Question Data

• canary (str): Anti-contamination string to prevent training data leakage
• description (str): Detailed context and background information for the question
• name (str): Unique question identifier for easy reference and tracking
• uuid (str): Universal unique identifier ensuring dataset integrity
• subfield (str): Specific chemistry/materials subcategory beyond the main configuration

📝 Question Content & Answers

• examples (list of dict): Question-answer pairs containing:
  • input (str): The complete question to be answered
  • target (str, optional): Expected answer for open-ended questions
  • target_scores (str, optional): Answer key for multiple-choice questions (1=correct, 0=incorrect)

🏷️ Categorization & Metadata

• keywords (list of str): Searchable tags including:
  • Difficulty levels: difficulty-basic, difficulty-intermediate, difficulty-advanced
  • Required skills: requires-knowledge, requires-reasoning, requires-calculation, requires-intuition
  • Topic-specific keywords for content discovery

📊 Evaluation Framework

• metrics (list of str): Available evaluation metrics:
  • Multiple-choice: multiple_choice_grade
  • Open-ended: exact_string_match, mae (Mean Absolute Error), mse (Mean Squared Error)
• preferred_score (str): Recommended primary evaluation metric for the question

🛠️ Tool Usage Indicators

• in_humansubset_w_tool (bool): Whether question requires computational tools
• in_humansubset_wo_tool (bool): Whether question can be solved without tools

🚀 Usage

Quick Start with ChemBench Engine

from chembench.evaluate import ChemBenchmark
from chembench.prompter import PrompterBuilder
from chembench.utils import enable_logging
from dotenv import load_dotenv

# Setup environment and logging
load_dotenv(".env")
enable_logging()

# Load the benchmark
benchmark = ChemBenchmark.from_huggingface()

# Configure your model
model = "openai/gpt-4"
prompter = PrompterBuilder.from_model_object(model=model)

# Run evaluation
results = benchmark.bench(prompter)

# Submit results to leaderboard
benchmark.submit(results)

For comprehensive documentation and advanced usage patterns, visit our documentation. 📚

🎯 Use Cases

ChemBench serves multiple purposes in the development and evaluation of AI systems for chemistry:

• 🤖 LLM Evaluation: Comprehensive assessment of large language models' chemistry knowledge and reasoning capabilities
• 📊 Model Comparison: Standardized benchmarking for comparing different AI models across chemistry domains
• � Research Development: Identifying strengths and weaknesses in AI systems to guide future research directions
• 🎓 Educational Assessment: Evaluating AI tutoring systems and educational tools for chemistry learning
• 🏢 Industry Applications: Testing AI systems before deployment in pharmaceutical, materials, and chemical industries
• 🧪 Expert Validation: Comparing AI performance against human chemistry experts and professionals

⚠️ Limitations & Considerations

• 🎯 Scope: Focused on chemistry and materials science; may not cover all specialized subdisciplines
• 📚 Knowledge Cutoff: Reflects current scientific understanding; new discoveries may not be included
• 🌍 Language: Primarily English content, limiting multilingual applications
• ⚖️ Complexity Distribution: While spanning basic to advanced levels, expert-level questions may be limited
• 🔄 Dynamic Field: Chemistry knowledge evolves rapidly; regular updates recommended
• 👥 Expert Bias: Reflects perspectives and knowledge of curating experts
• 📊 Evaluation Metrics: Some nuanced chemical reasoning may not be fully captured by current metrics

🛠️ Data Processing Pipeline

ChemBench follows a rigorous curation and validation process:

1. 👨‍🔬 Expert Curation: Questions created by chemistry and materials science professionals
2. 📚 Content Review: Multi-expert validation of question accuracy and relevance
3. 🏷️ Metadata Assignment: Comprehensive tagging with keywords, difficulty levels, and skill requirements
4. ⚖️ Quality Control: Systematic review for clarity, accuracy, and appropriate difficulty distribution
5. 🔧 Format Standardization: Consistent JSON structure across all chemistry domains
6. ✅ Validation Testing: Pilot testing with human experts to ensure question quality
7. 📊 Statistical Analysis: Distribution analysis to ensure balanced representation across topics

🏗️ ChemBench Framework

This dataset is designed to work seamlessly with the ChemBench evaluation engine, providing:

🚀 Key Features

• 🔄 Automated Evaluation: Streamlined assessment pipeline for various model types
• 📈 Leaderboard Integration: Direct submission to public performance leaderboards
• 🛠️ Tool Integration: Support for models with and without computational tool access
• 📊 Comprehensive Metrics: Multiple evaluation approaches for different question types
• 🌐 Community Driven: Open-source framework encouraging community contributions

💡 Flexibility

While optimized for the ChemBench engine, the dataset can be adapted for use with any benchmarking framework, making it accessible to the broader AI research community.

📄 Citation

If you use ChemBench in your research, please cite:

@article{alampara2024probing,
  title   = {Probing the limitations of multimodal language models for chemistry and materials research},
  author  = {Nawaf Alampara and Mara Schilling-Wilhelmi and Martiño Ríos-García and Indrajeet Mandal and Pranav Khetarpal and Hargun Singh Grover and N. M. Anoop Krishnan and Kevin Maik Jablonka},
  year    = {2024},
  journal = {arXiv preprint arXiv: 2411.16955}
}

👥 Contact & Support

• 📄 Paper: Nature Chemistry Publication
• 🌐 Website: MaCBench Project
• 🤗 Dataset: Hugging Face
• 🏆 Leaderboard: Model Performance Rankings
• 💻 Code: GitHub Repository
• 📚 Documentation: Full Documentation
• ❓ Issues: Report problems or ask questions via the Hugging Face dataset page or GitHub repository

---