Overview

This repository contains a lightweight library for evaluating language models. We are open sourcing it so we can be transparent about the accuracy numbers we're publishing alongside our latest models.

Benchmark Results

Background

Evals are sensitive to prompting, and there's significant variation in the formulations used in recent publications and libraries. Some use few-shot prompts or role playing prompts ("You are an expert software programmer..."). These approaches are carryovers from evaluating base models (rather than instruction/chat-tuned models) and from models that were worse at following instructions.

For this library, we are emphasizing the zero-shot, chain-of-thought setting, with simple instructions like "Solve the following multiple choice problem". We believe that this prompting technique is a better reflection of the models' performance in realistic usage.

We will not be actively maintaining this repository and monitoring PRs and Issues. In particular, we're not accepting new evals. Here are the changes we might accept.

• Bug fixes (hopefully not needed!)
• Adding adapters for new models
• Adding new rows to the table below with eval results, given new models and new system prompts.

This repository is NOT intended as a replacement for https://github.com/openai/evals, which is designed to be a comprehensive collection of a large number of evals.

Evals

This repository currently contains the following evals:

• MMLU: Measuring Massive Multitask Language Understanding, reference: https://arxiv.org/abs/2009.03300, https://github.com/hendrycks/test, MIT License
• MATH: Measuring Mathematical Problem Solving With the MATH Dataset, reference: https://arxiv.org/abs/2103.03874, https://github.com/hendrycks/math, MIT License
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark, reference: https://arxiv.org/abs/2311.12022, https://github.com/idavidrein/gpqa/, MIT License
• DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs, reference: https://arxiv.org/abs/1903.00161, https://allenai.org/data/drop, Apache License 2.0
• MGSM: Multilingual Grade School Math Benchmark (MGSM), Language Models are Multilingual Chain-of-Thought Reasoners, reference: https://arxiv.org/abs/2210.03057, https://github.com/google-research/url-nlp, Creative Commons Attribution 4.0 International Public License (CC-BY)
• HumanEval: Evaluating Large Language Models Trained on Code, reference https://arxiv.org/abs/2107.03374, https://github.com/openai/human-eval, MIT License

Samplers

We have implemented sampling interfaces for the following language model APIs:

• OpenAI: https://platform.openai.com/docs/overview
• Claude: https://www.anthropic.com/api

Make sure to set the *_API_KEY environment variables before using these APIs.

Setup

Due to the optional dependencies, we're not providing a unified setup mechanism. Instead, we're providing instructions for each eval and sampler.

For HumanEval (python programming)

git clone https://github.com/openai/human-eval
pip install -e human-eval

For the OpenAI API:

pip install openai

For the Anthropic API:

pip install anthropic

Running the evals

python -m LLM-Calibration-Study.simple_evals --list-models

This will list all the models that you can evaluate.

To run the evaluations, you can use the following command:

python -m LLM-Calibration-Study.simple_evals --model <model_name> --examples <num_examples>

This will launch evaluations through the OpenAI API.

Notes

Legal Stuff

By contributing to evals, you are agreeing to make your evaluation logic and data under the same MIT license as this repository. You must have adequate rights to upload any data used in an eval. OpenAI reserves the right to use this data in future service improvements to our product. Contributions to OpenAI evals will be subject to our usual Usage Policies: https://platform.openai.com/docs/usage-policies.

Ivan's Additions

The model can now benchmark MMLU, GPQA and SimpleQA with the following confidence extraction methods: verbal_vanilla, verbal_cot, empirical_semantic, single_generation, faithfulness (faithful response uncertainty).

To initiate benchmarking, run:

python -m LLM-Calibration-Study.simple_evals --model [MODEL NAME] [--examples ANY NUMBER OF EXAMPLES] [--regenerate] --benchmark [BENCHMARK] --conf_mode [CONFIDENCE EXTRACTION METHOD]

[BENCHMARK] is one of mmlu, gpqa, simpleqa

[CONFIDENCE EXTRACTION METHOD] is one of verbal_vanilla, verbal_cot, empirical_semantic, single_generation, faithfulness

--regenerate is an optional argument. When used, the program saves (pickles) previously generated responses (including questions and responses) in the \cache directory. When a test is run again, the program first searches for the existence of the corresponding cache. If the search fails, the program then resamples as usual. When --regenerate is not specified, responses are sampled regardless of the existence of the corresponding cache but the results will be cached. The cache is named by LLM-Calibration-Study/cache/[BENCHMARK]_[MODEL NAME]_[CONFIDENCE EXTRACTION METHOD]_{NUMBER OF EXAMPLES}_{NUMBER OF REPEATS}. Thus, any changes will be treated as a new run.

Example usage:

The following uses meta-llama/Llama-3.3-70B-Instruct-Turbo-Free to run 3 examples against SimpleQA with the empirical semantic confidence extraction method. When the same command runs the second time or more, the program won't resample its responses. python -m LLM-Calibration-Study.simple_evals --model meta-llama/Llama-3.3-70B-Instruct-Turbo-Free --examples 3 --benchmark simpleqa --conf_mode logit_perplexity --regenerate

Evaluation results are saved in an HTML file (now also includes confidence extraction mode and log probs) along with a JSON as before.

https://cloud.google.com/sdk/docs/install-sdk#before-you-begin

Footnotes

1. We believe these evals are saturated for our newer models, but are reporting them for completeness. ↩ ↩²

2. Includes an answer regex tweak for GPQA benchmark. ↩

3. o-series models do not support using a system prompt. ↩ ↩²

4. The default reasoning level for o3-mini is "medium". ↩

5. Low scores in non-latin languages like be, tn, th. ↩

6. For o1 models, we evaluate on MATH-500, which is a newer, IID version of MATH. ↩

7. assistant system message in OpenAI API doc: "You are a helpful assistant." . ↩