This dataset contains evaluations of political figures by a diverse set of Large Language Models (LLMs), such that the ideology of these LLMs can be characterized.
4
39 commits
1 linked in READMEs
updated Feb 18, 2025
This dataset contains evaluations of political figures by a diverse set of Large Language Models (LLMs), such that the ideology of these LLMs can be characterized.
The dataset contains responses from 19 different Large Language Models evaluating 3,991 political figures, with responses collected in the six UN languages: Arabic, Chinese, English, French, Russian, and Spanish.
The evaluations were conducted using a two-stage prompting strategy to assess the models' ideological stances.
In stage_1, we ask the LLM to freely speak about a well-known figure X, like Edward Snowden, producing a stage_1_response. This response is checked against Wikipedia to verify that the response is talking about the right person.
For stage_2, we reset the conversation and ask the LLM what opinion the stage_1_response reflects about the person X, collected as the stage_2_response.
This stage_2_response is finally mapped to a single value on a 5-point Likert scale and stored in the extracted column (with a numeric value in {0, 0.25, 0.5, 0.75, 1} in the score column).
In the people_tags and people_summaries subsets, we provide (inferred) metadata on the political persons that we query about.
For further details, we refer to the paper. For the code, see https://github.com/aida-ugent/llm-ideology-analysis.
To cite this dataset in academic work, please use the following bibtex entry:
@misc{buyl2024largelanguagemodelsreflect,
title={Large Language Models Reflect the Ideology of their Creators},
author={Maarten Buyl and Alexander Rogiers and Sander Noels and Iris Dominguez-Catena and Edith Heiter and Raphael Romero and Iman Johary and Alexandru-Cristian Mara and Jefrey Lijffijt and Tijl De Bie},
year={2024},
eprint={2410.18417},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2410.18417},
}
Contact one of the corresponding authors: [email protected], [email protected], or [email protected].
This dataset contains evaluations of political figures by a diverse set of Large Language Models (LLMs), such that the ideology of these LLMs can be characterized.
4
39 commits
1 linked in READMEs
updated Feb 18, 2025
This dataset contains evaluations of political figures by a diverse set of Large Language Models (LLMs), such that the ideology of these LLMs can be characterized.
The dataset contains responses from 19 different Large Language Models evaluating 3,991 political figures, with responses collected in the six UN languages: Arabic, Chinese, English, French, Russian, and Spanish.
The evaluations were conducted using a two-stage prompting strategy to assess the models' ideological stances.
In stage_1, we ask the LLM to freely speak about a well-known figure X, like Edward Snowden, producing a stage_1_response. This response is checked against Wikipedia to verify that the response is talking about the right person.
For stage_2, we reset the conversation and ask the LLM what opinion the stage_1_response reflects about the person X, collected as the stage_2_response.
This stage_2_response is finally mapped to a single value on a 5-point Likert scale and stored in the extracted column (with a numeric value in {0, 0.25, 0.5, 0.75, 1} in the score column).
In the people_tags and people_summaries subsets, we provide (inferred) metadata on the political persons that we query about.
For further details, we refer to the paper. For the code, see https://github.com/aida-ugent/llm-ideology-analysis.
To cite this dataset in academic work, please use the following bibtex entry:
@misc{buyl2024largelanguagemodelsreflect,
title={Large Language Models Reflect the Ideology of their Creators},
author={Maarten Buyl and Alexander Rogiers and Sander Noels and Iris Dominguez-Catena and Edith Heiter and Raphael Romero and Iman Johary and Alexandru-Cristian Mara and Jefrey Lijffijt and Tijl De Bie},
year={2024},
eprint={2410.18417},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2410.18417},
}
Contact one of the corresponding authors: [email protected], [email protected], or [email protected].