ZeroEval: A Unified Framework for Evaluating Language Models

ZeroEval is a simple unified framework for evaluating (large) language models on various tasks. This repository aims to evaluate instruction-tuned LLMs for their zero-shot performance on various reasoning tasks such as MMLU and GSM. We evaluate LLMs with a unified setup by controlling the factors such as prompting, sampling, output parsing, etc. In ZeroEval, we perform zero-shot prompting, and instruct LM to output both reasoning and answer in a json-formatted output. We are actively adding new tasks. Contributions are welcome!

• Leaderboard: https://hf.co/spaces/allenai/ZeroEval
• X post

Todo

• Support new tasks (GPPA, AIME, etc.)
• Prefix-prefill for open models such that the parsing is easier
• Add other formatting options (e.g. markup language instead of json, etc.)

Installation

conda create -n zeroeval python=3.10
conda activate zeroeval
# pip install vllm -U # pip install -e vllm
pip install vllm -U
pip install -r requirements.txt
# export HF_HOME=/path/to/your/custom/cache_dir/

Tasks

• MMLU-redux (-d mmlu-redux)

• ZebraLogic (-d zebra-grid)

• CRUX (-d crux)

• MATH (Level 5) (-d math-l5)

• GSM8K (-d gsm)

• More tasks will be added soon. (e.g., ARC, MMLU-Pro, etc.)

Usage

zero_eval_local.sh and zero_eval_api.sh are the two main scripts to run the evaluation.

Examples

• bash zero_eval_local.sh -d mmlu-redux -m meta-llama/Meta-Llama-3-8B-Instruct -p Meta-Llama-3-8B-Instruct -s 4 (Run Llama-3-8B-Instruct with greedy decoding on mmlu-redux)

• bash zero_eval_api.sh -d gsm -f openai -m openai/gpt-4o-mini-2024-07-18 -p gpt-4o-mini-2024-07-18 -s 8 (Run gpt-4o-mini with greedy decoding on gsm)

• bash zero_eval_api.sh -d zebra-grid -f openai -m deepseek-chat -p deepseek-chat -s 8 (Run deepseek-chat via openai style api, with greedy decoding on zebra-grid)

More examples can be found in the scripts folder, e.g., the scripts/_MMLU_redux.md and scripts/_GSM.md files as well as scripts/local/crux.sh.

Arguments

Results

🚨 View results on our Leaderboard: https://hf.co/spaces/allenai/ZeroEval

• MMLU-Redux: python src/evaluation/mcqa_eval.py mmlu-redux --> Full results
• GSM/MATH-L5: python src/evaluation/math_eval.py math-l5/gsm --> Full results
• ZebraLogic: python src/evaluation/zebra_grid_eval.py --> Full results and Leaderboard
• CRUX: python src/evaluation/crux_eval.py --> Full results
• All: python src/evaluation/summarize.py --> Full results ⬇️

Citation

If you find ZeroEval useful, please cite it as follows in your publication:

@software{Lin_ZeroEval_A_Unified_2024,
    author = {Lin, Bill Yuchen},
    month = jul,
    title = {{ZeroEval: A Unified Framework for Evaluating Language Models}},
    url = {https://github.com/WildEval/ZeroEval},
    year = {2024}
}

Star History