Taming Knowledge Conflicts in Language Models

This repository contains the code and data of the ICML 25 Spotlight Paper Taming Knowledge Conflicts in Language Models. The code is now still being updated.

Requirements

To install the dependecies in a new conda environment, run

$ conda create --name <env> --file conda_env/requirement.txt

The main requirements are Transformers==4.48.0 and nnsight==0.3.6.

How to run experiments

Scripts can be found in /src/scripts.

Dataset

To obtain your own dataset splits, please follow the procedures below:

1. run python src/generate_dataset_split.py to obtain the per-category dataset split.
2. run CUDA_VISIBLE_DEVICES=device_num python src/dataset.py --model your_model --dataset category_ds to get model-dataset split.

We provide the Gemma model dataset for reproducibility purposes.

Citations

arXiv:

@article{li2025taming,
  title={Taming Knowledge Conflicts in Language Models},
  author={Li, Gaotang and Chen, Yuzhong and Tong, Hanghang},
  journal={arXiv preprint arXiv:2503.10996},
  year={2025}
}

conference:

@inproceedings{
li2025taming,
title={Taming Knowledge Conflicts in Language Models},
author={Gaotang Li and Yuzhong Chen and Hanghang Tong},
booktitle={Forty-second International Conference on Machine Learning},
year={2025},
url={https://openreview.net/forum?id=0cEZyhHEks}
}