Testing Language Models on a Held-Out High School National Finals Exam

When xAI recently released Grok-1, they evaluated it on the 2023 Hungarian national high school finals in mathematics, which was published after the training data cutoff for all the models in their evaluation. While MATH and GSM8k are the standard benchmarks for evaluating the mathematical abilities of large language models, there are risks that modern models overfit to these datasets, either from training directly on the test sets or from tuning the model hyperparameters to maximize test set performance. By evaluating on a truly held out test set, we can better guage the mathematical performance of these models.

We evaluate on the 2023 Hungarian national high school finals in mathematics and grade by hand using the provided rubric. All model solutions were graded by myself over the course of one day. Model solutions were sampled using temperature 0.1.

For base models such as Code Llama, Llemma, and Mistral-7B, a 5-shot prompt was used. For instruction tuned models, we used the default prompt template for that model.

Results

Note: In an earlier version of the LaTeX transcription of the exam, question 14a had incorrect formatting and question 14b did not contain all the required information to solve the problem. These issues have been fixed and the numbers are updated.

Observations

1. Plotting GSM8k performance versus performance on the exam, we can see clear evidence that several models overfit to the benchmark.
2. Despite claiming that OpenChat 3.5 is competitive with Grok-1, it only gets around half the score on the held-out exam, indicating that is simply overfits to evaluations.
3. Llemma 34B is competitive with GPT-3.5 Turbo on the held-out exam. Further instruction tuning Llemma 34B should give even greater performance.

Solutions

Please find model solutions and corresponding grades in the solutions folder.

Running the Evaluation

To run the evaluation, run the following command:

python run_exam.py --model EleutherAI/llemma_34b --exam test/exam.csv --prompt few_shot

Notes on Grading

There are a few problems which either require creating or reading a figure. For these problems, I graded the model solutions as incorrect. In the future when models have these abilities, this should be changed.

Citation

To cite this article, use the following citation:

@misc{testing_language_models_on_a_held_out_high_school_national_finals_exam,
  title={Testing Language Models on a Held-Out High School National Finals Exam},
  author={Keiran Paster},
  howpublished={\url{https://huggingface.co/datasets/keirp/hungarian_national_hs_finals_exam}},
  journal = {HuggingFace repository},
  year={2023},
}