Data and code for the paper "Leveraging In-Context Learning for Political Bias Testing of LLMs" (ACL 2025)

Installation

• Prerequisites: Python >= 3.8, PyTorch
• pip install -r requirements.txt

Data

Survey Data

The data/questionnaire directory contains the following files:

• questions.jsonl: The 60 in-context questions (type "context") and the 7 target questions (type "target")
• human_respondents.jsonl: The answers by the 192 respondents, ranging from 0 (strongly disagree) to 100 (strongly agree). One line corresponds to one respondent.

© Smartvote. All rights reserved. Released for academic research purposes only.

Prompts

The files of prompts that we generate from the survey data are too large to be fully included in this repository. Instead, we provide the following:

• The Python scripts scripts/preprocess_data/generate_prompts.py, etc., which generate the prompts from the survey data and store them in the data/prompts directory
• Sample files in data/prompts containing one prompt as an example

Model Outputs

Outputs for the models Llama 3.1 8B (base/instruct), 70B (base/instruct), 405B (instruct), OLMo 7B (base/instruct), and GPT-3.5 are included in the data/model_outputs directory.

Reproducing the results in the paper

Calculate personalization accuracy and bias and create response overview table (Table 1)

python -m scripts.create_tables.calculate_accuracy_and_bias

Table 2 (personalization accuracy and bias: base vs instruct) and Table 3 (personalization accuracy and bias: base vs instruct: comparison by model size)

python -m scripts.create_tables.acc_bias_base_instruct

python -m scripts.create_tables.acc_bias_size

Table 4 (average bias variability) and Table 6 (average bias variability per question)

python -m scripts.create_tables.calculate_variability

Figure 3 (token distribution example)

See the Jupyter notebook scripts/visualize_distributions/example.ipynb

Figures 4 and 7 (correlation between personalization accuracy and bias) and Figure 5 (political stance scores)

Installing dependencies: Rscript -e 'install.packages(c("viridis", "dplyr", "ggrepel", "paletteer", "tidyr"), repos="https://cloud.r-project.org")'

Rscript --vanilla scripts/create_tables/plots.R

Figure 6 (distribution of human answers)

python -m scripts.create_tables.plot_human_answer_distribution

Recreating the model outputs

Llama 3.1 8 & 70B / OLMo

python -m scripts.run_models.run_hf_llm <model_name> <prompt_jsonl_path>

• <model_name>: meta-llama/Meta-Llama-3.1-8B / meta-llama/Meta-Llama-3.1-8B-Instruct / meta-llama/Meta-Llama-3.1-70B / meta-llama/Meta-Llama-3.1-70B-Instruct / allenai/OLMo-7B-hf / allenai/OLMo-7B-Instruct-hf
• <prompt_jsonl_path>: Path to one of the JSONL files in data/prompts

Llama 3.1 405B

• python -m scripts.run_models.run_together_llm meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo <prompt_jsonl_path>

GPT-3.5

• python -m scripts.run_models.run_gpt_batch <prompt_jsonl_path>
• python -m scripts.run_models.gpt_batch_to_csv <batch_output_path>

Citation

@inproceedings{haller-et-al-2025-leveraging,
    title = "Leveraging In-Context Learning for Political Bias Testing of LLMs",
    author = "Haller, Patrick  and
      Vamvas, Jannis  and
      Sennrich, Rico  and
      J{\"a}ger, Lena Ann",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics"
}