Cognitive profiles of language models

This repository contains the code to reproduce the experiments of the paper "Language models emulate certain cognitive profiles: An investigation of how predictability measures interact with individual differences" (ACL 2024).

Code

Estimate and extract surprisal and entropy measures

python src/experiments.py

Analysis scripts

Rscript --vanilla src/analyses.R

Citation

Patrick Haller, Lena Bolliger, and Lena Jäger. 2024. Language models emulate certain cognitive profiles: An investigation of how predictability measures interact with individual differences. In Findings of the Association for Computational Linguistics ACL 2024, pages 7878–7892, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.