WILA-popQA

This repository contains the code and multilingual dataset for our LREC 2026 workshop paper "A Wikidata-Based Framework to Measure Cross-Lingual Bias in Multilingual Large Language Models." The interdisciplinary project is focused on understanding cultural biases in multilingual LLMs. It explores how language alignment affects the performance of LLMs, particularly when answering factual questions about cultural entities such as authors, historical figures, and traditional practices. By leveraging Wikidata Knowledge Graphs, this project aims to provide a thorough analysis of the capabilities and limitations of multilingual LLMs.

Directory Structure

• data_collection/ — scripts to retrieve and process Wikidata entities and generate prompts
  • data/ — intermediate and processed dataset files
• evaluation/
  • evaluation_scripts/ — evaluation scripts, organized per property
  • reports/ — report generation scripts and output reports
• inference/ — scripts for running batch inference on LLMs

Installation and Setup

Requires Python 3.11 or higher, uv and Docker. Large data files (experiments, evaluations, prompts) are versioned with DVC and stored on Google Drive. A public version of our data is available here and is required to reproduce the pipeline.

1. Clone the repository.

2. To install dependencies run:

uv sync

3. Download our data via this link. The data includes the processed dataset, prompts, and evaluation results.

4. Unzip the downloaded folder in the root directory of the cloned repository

5. (optional) Follow the steps outlined in the DVC documentation to set up your own DVC remote.

6. (optional) To push the files to your DVC remote:

uv run dvc push

Usage

To reproduce the full pipeline, run:

docker run -p 1234:1234 --name qendpoint-wikidata --env MEM_SIZE=6G qacompany/qendpoint-wikidata
uv run dvc repro

Individual stages can be run with:

uv run dvc repro <stage_name>

Please note that the stages get_entities and get_data require running the docker command above.

To visualize the dependency graph run:

uv run dvc dag

See dvc.yaml for the full pipeline definition and available stages.

For Contributors

To download the latest file versions and run the pipeline with the Google Drive remote storage, access credentials as well as the Google folder ID are required. Configure the DVC remote with:

uv run dvc remote modify --local gdrive url gdrive://<FOLDER_ID>
uv run dvc remote modify --local gdrive gdrive_client_id 'client-id'
uv run dvc remote modify --local gdrive gdrive_client_secret 'client-secret'
                       

To pull the latest data from the remote, run:

uv run dvc pull                             

License

This project is licensed under the MIT License. Refer to the LICENSE file for more details.

Contact

If you have any questions, feel free to contact us via GitHub or reach out to the contributors directly.

Citation

If you use the WILA-PopQA dataset or this codebase, please cite our paper:

@inproceedings{iferroudjene-etal-2026-wikidata,
  title = {A Wikidata-Based Framework to Measure Cross-Lingual Bias in Multilingual Large Language Models },
  author = {Iferroudjene, Mouloud and Poggel, Lisa and Schimmenti, Andrea and Yang, Duo and Shivashankar, Kanchan and Kalo, Jan-Christoph and Boscariol, Marta},
  booktitle = {Proceedings of the Knowledge Graphs and Large Language Models Workshop (KG-LLM) @ LREC26},
  month = {May},
  year = {2026},
  pages = {190--209},
  address = {Palma, Mallorca, Spain},
  publisher = {European Language Resources Association (ELRA)},
  editor = {Sérasset, Gilles and Gkirtzou, Katerina and Cochez, Michael and Kalo, Jan-Christoph},
  doi = {10.63317/29b354pejyrt},
  abstract = {Multilingual large language models (LLMs) are increasingly used for factual question answering, yet their accuracy varies across languages in ways that are difficult to interpret. A central challenge is that many multilingual probing benchmarks conflate multiple factors: the language used to ask the question, the cultural-linguistic context of the entities being queried, and the popularity skew of entities. In our paper, we disentangle these factors by asking: (i) how strongly does the Language of the Question (LoQ) affect factual recall, (ii) does matching LoQ to an entity-associated Language of the Entity (LoE) improve performance, and (iii) do these effects persist when entity popularity is controlled. To this end, we introduce WILA-PopQA, a new Wikidata-grounded benchmark spanning 9 languages with matched popularity profiles, and probe 12 open-weight models of varying sizes and architectures under aligned and misaligned LoQ–LoE conditions. We evaluate models’ answers to 4 types of questions about entity biographical properties in all selected languages. Results show that LoQ is the dominant source of variation. LoQ–LoE alignment does not consistently yield the highest accuracy, and performance depends on the property being asked. These results suggest that prompt language is an actionable experimental factor for multilingual factual evaluation.}
}

The dataset is also available on Zenodo:

@dataset{wila_popqa_2026,
  title     = {WILA-PopQA: Popularity-matched Multilingual Wikidata QA dataset},
  author    = {Iferroudjene, Mouloud and Poggel, Lisa and Schimmenti, Andrea and Yang, Duo and Shivashankar, Kanchan and Kalo, Jan-Christoph and Boscariol, Marta},
  year      = {2026},
  month     = mar,
  publisher = {Zenodo},
  version   = {v2},
  doi       = {10.5281/zenodo.19249706},
  url       = {https://zenodo.org/records/19249706}
}